Title: A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Visual Reinforcement Learning

Generalization
Abstract: In the rapidly evolving domain of vision-based deep reinforcement learning (RL), a pivotal challenge is to achieve generalization capability to dynamic environmental changes reflected in visual observations.
Our work delves into the intricacies of this problem, identifying two key issues that appear in previous approaches for visual RL generalization: (i) imbalanced saliency and (ii) observational overfitting.
Imbalanced saliency is a phenomenon where an RL agent disproportionately identifies salient features across consecutive frames in a frame stack. 
Observational overfitting occurs when the agent focuses on certain background regions rather than task-relevant objects.
To address these challenges, we present a simple yet effective framework for generalization in visual RL (SimGRL) under dynamic scene perturbations.
First, to mitigate the imbalanced saliency problem, we introduce an architectural modification to the image encoder to stack frames at the feature level rather than the image level.
Simultaneously, to alleviate the observational overfitting problem, we propose a novel technique called shifted random overlay augmentation, which is specifically designed to learn robust representations capable of effectively handling dynamic visual scenes.
Extensive experiments demonstrate the superior generalization capability of SimGRL, achieving state-of-the-art performance in benchmarks including the DeepMind Control Suite.

A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations

Abstract

In the rapidly evolving domain of vision-based deep reinforcement learning (RL), a pivotal challenge is to achieve generalization capability to dynamic environmental changes reflected in visual observations. Our work delves into the intricacies of this problem, identifying two key issues that appear in previous approaches for visual RL generalization: (i) imbalanced saliency and (ii) observational overfitting. Imbalanced saliency is a phenomenon where an RL agent disproportionately identifies salient features across consecutive frames in a frame stack. Observational overfitting occurs when the agent focuses on certain background regions rather than task-relevant objects. To address these challenges, we present a simple yet effective framework for generalization in visual RL (SimGRL) under dynamic scene perturbations. First, to mitigate the imbalanced saliency problem, we introduce an architectural modification to the image encoder to stack frames at the feature level rather than the image level. Simultaneously, to alleviate the observational overfitting problem, we propose a novel technique called shifted random overlay augmentation, which is specifically designed to learn robust representations capable of effectively handling dynamic visual scenes. Extensive experiments demonstrate the superior generalization capability of SimGRL, achieving state-of-the-art performance in benchmarks including the DeepMind Control Suite. 1

1 Introduction

Deep reinforcement learning (RL) utilizing visual observations has achieved remarkable success across diverse domains, including robotic manipulation [21], video games [25, 1, 38], and autonomous navigation [24, 48]. However, acquiring generalizable RL policies across diverse environments remains challenging, mainly due to overfitting [44] in the high-dimensional observation space [32]. To obtain robust policies invariant to visual perturbations, a variety of approaches based on domain randomization [36, 28] and data augmentation [31, 19, 40, 29, 13, 12] have been widely proposed. These approaches operate under the assumption that exposing an agent to various augmentations during the training phase enhances its adaptability to unseen domains. Despite the encouraging results, performance still falls behind in challenging environments with dynamically changing backgrounds.

(a) DMControl Generalization Performance

(b) Imbalanced Saliency

(c) Observational Overfitting

DMControl-GBVideoBenchmarks

StackedFrames

StackedFrames

SimGRL

t-2

t-1

t

t-2

t-1

t

CG2A

EAR

SGQN

TLDA

SVEA

Observation

Observation

SODA

DrQ

RAD

SAC

Video Easy

Video Hard

0

100

200

300

400

500

600

700

800

s O Mp

s O Mp

AverageScore

Fig. 1(a) shows the notable performance degradation of existing approaches when comparing 'Video Hard' scenes with dynamically changing backgrounds to less dynamic 'Video Easy' scenes in the DeepMind Control Suite-Generalization Benchmark (DMControl-GB) [13].

Using a gradient-based attribution mask M ρ [2], we first investigate the causes of the degradation in generalization in such challenging environments by examining salient regions across consecutive stacked frames used as an RL input. Based on our analysis, we empirically identified two phenomena, highlighting them as key causes of performance degradation: (i) what we refer to as imbalanced saliency and (ii) observational overfitting [32]. In the DMControl [35, 37], Fig. 1(b) depicts an example of the imbalanced saliency in a 'Cartpole, Swingup' task, where salient features across all stacked frames are biased to the regions corresponding to the task objects in the latest two frames. Fig. 1(c) shows an example of the observational overfitting that occurs in a 'Cheetah, Run' task. In this case, the RL agent misidentifies the ground as more salient than the 'Cheetah' object. These two problems contribute to overfitting to the training environment, making generalization of the RL agent even more challenging.

In image-based RL, the effectiveness of regularization has been demonstrated in numerous studies. DrQ [40] and RAD [19], each employing random shift and random crop augmentations as data regularization on input images, achieved remarkable performance improvements over vanilla SAC [8]. Furthermore, [32] exhibited that regularization effects from architectural modifications such as overparameterization or residual connections can help avoid observational overfitting and reduce the generalization gap.

In this context of research, we introduce a Sim ple yet effective framework for G eneralization in visual RL (SimGRL) under dynamic scene perturbations. Firstly, we empirically found the image-level frame stack, commonly used in traditional vision-based model-free RLapproaches [25, 18, 41, 40, 19], to be the main factor causing the imbalanced saliency problem. To address this issue, we propose a simple architectural modification of an image encoder, which employs a feature-level frame stack instead of the image level. This involves extracting individual feature maps for each frame from the shallow layers of the encoder, stacking them along channels, and encoding the stacked feature maps through the remaining layers. Since this approach considers solely individual frames during the initial encoding, the agent is trained to focus on spatially salient features in each consecutive frame that are essential for a given task, alleviating the imbalanced saliency problem of Fig. 1(b). Secondly, to address the observational overfitting problem by encouraging the agent to focus on the task object rather than backgrounds, we propose a new data augmentation called shifted random overlay, which is a modified version of random overlay augmentation [13]. This augmentation directly injects background dynamics irrelevant to the given task into the agent during the training phase by interpolating natural images moving in a random direction on each frame. By enabling the agent to implicitly learn to ignore task-irrelevant backgrounds and focus on task-relevant pixels, this approach alleviates the observational overfitting problem of Fig. 1(c). Furthermore, this augmentation enables the agent to adapt to test environments with dynamic perturbations in the surrounding backgrounds.

We verify that these strategies can remarkably improve generalization in challenging environments, achieving state-of-the-art performance without using any additional auxiliary losses or networks.

Furthermore, utilizing the attribution mask M ρ [2], we introduce novel metrics, called T askId entification (TID) metrics, consisting of TID score and TID variance, to quantitatively evaluate the discrimination ability on salient regions. With the proposed metrics, we quantitatively analyze the two problematic phenomena in the existing approaches, demonstrating the excellent discrimination ability of SimGRL. Moreover, we show a tendency of positive correlation between the TID score and RL generalization performance, emphasizing the importance of accurately identifying salient features in input images.

Our contributions include the following aspects:

By utilizing gradient-based attribution masks, we highlight the two core issues of imbalanced saliency and observational overfitting, which hinder the generalization of visual RL for most model-free RL settings. Additionally, we propose TID metrics to measure the discrimination ability of an RL agent on task objects, providing insights into these issues.

To address these problems, we propose architectural and data regularization methods through a modification to an encoder structure and an introduction of new data augmentation.

We achieve state-of-the-art performances across video benchmarks of DMControl-GB [13], DistractingCS [34], and robotic manipulation tasks [14].

2 Background

Visual RL and Generalization We consider a partially observable Markov decision process (POMDP) problem M = ( S , O , A , P , r , γ ) , where S is state space, O is observation space, A is action space, P : S × A → S is the transition function that defines the conditional probability distribution P ( s t +1 | s t , a t ) over next states given a state s t ∈ S and an action a t ∈ A taken at time t , r : S × A → R is a reward function, and γ ∈ [0 , 1) is a discount factor. In the POMDP problem, such as the visual RL, because only the high-dimensional observations o t ∈ O can be observable [15], a state is defined as a sequence of the k consecutive image frames s t = ( o t -k +1 , ..., o t -1 , o t ) [25]. Without loss of generality, we will set k = 3 for the sake of notational simplicity. RL aims to learn a policy π : S → A that maximizes the expected sum of discounted rewards E π [ ∑ T t =0 γ t r ( s t , r t )] . In this work, we focus on the generalization problem to POMDPs ̂ M = ( ̂ S , ̂ O , A , P , r , γ ) , where the states ˆ s ∈ ̂ S are constructed from the perturbed observations ˆ o ∈ ̂ O , and a POMDP ̂ M is sampled from the space of POMDPs M , ̂ M ∼ M .

Deep Q-Learning and Soft Actor-Critic Deep Q-learning [25] is a common model-free RL algorithm that aims to learn a parameterized state-action value function Q θ ( s t , a t ) , where θ is the deep neural network parameters of the Q-function and action is greedily selected as the one with the maximum value a t = argmax a Q θ ( s t , a ) at time t . The training of the Q-function is achieved by minimizing a mean squared error of the Bellman residuals:

where ˆ θ is the parameters of the target Q network Q tgt and B is a replay buffer. To increase the stability, the parameter of the target Q network is slowly updated by the exponential moving average (EMA) ˆ θ = λθ +(1 -λ ) ˆ θ with λ ≪ 1 [22]. For the continuous action space, rather than the greedy sampling, a parameterized actor π ϕ ( s t ) is employed as the policy, where ϕ is the neural network parameters of the actor. Soft Actor-Critic (SAC) [8] is a common actor-critic algorithm with a state-action value Q θ ( s t , a t ) and a stochastic policy π ϕ ( a t | s t ) , and a temperature parameter α , which aims to optimize a γ -discounted maximum-entropy objective [49].

In visual RL, a parameterized encoder f θ : R C × H × W → R d is employed to compress the highdimensional image inputs and shared by both the actor and critic, where d is a dimension of the encoded feature. Consistent with the prior visual RL approaches [41, 18, 19, 40, 12], we jointly train the encoder with the critic and freeze it during actor updates, where the encoder learns representations for RL tasks by the critic loss. For notational simplicity, we denote all parameters updated by the critic loss as θ and the actor parameters as ϕ .

Gradient-based Attribution Mask An attribution map, also known as a saliency map, is designed to visualize the salient pixels in input images for given tasks. A common approach to computing the attribution map is a gradient-based method [30, 3], which indicates how sensitive the task prediction is to perturbations in the input pixels. Consistent with SGQN [2], we employ guided backpropagation [33] to compute the attribution map M ( Q θ , s t , a t ) = ∂Q θ ( s t ,a t ) ∂s t , where s t and M ( Q θ , s t , a t ) ∈ R C × H × W . Then, the binarized attribution mask M ρ ( Q θ , s t , a t ) , referred to as ρ -quantile attribution mask, is computed by thresholding M ( Q θ , s t , a t ) by the ρ -quantile, where M ρ ( Q θ , s t , a t ) ( i,j,k ) = 1 if the pixel value of M ( Q θ , s t , a t ) ( i,j,k ) belongs to the ρ -quantile of highest values for M ( Q θ , s t , a t ) , and 0 otherwise. We use this attribution mask to investigate the cause of the degradation of the generalization performance of existing methods in challenging environments such as Video Hard [13]. Furthermore, to justify our analysis, we will introduce new metrics that quantitatively measure the ability to identify salient pixels in Section 4.4. Note that we leverage the attribution mask only for analysis without involving it in the training process.

3 Pitfalls within Conventional Practices in Visual RL Generalization

The generalization ability of RL agents is often degenerated in challenging environments characterized by dynamic perturbations and different structures compared to the training environment. In this section, we investigate the causes of the overfitting to the training environment observed in existing visual RL methods on the DMControl Generalization Benchmark (DMControl-GB) [13].

Conventional practices used to train the visual RL agent for generalization [13, 2, 42, 4] include frame stack and data augmentation. First, to reflect the temporal structure of the input state, Q-value is predicted using stacked frames [25, 18, 41, 40, 19] instead of a single image as follows:

where θ is a neural network parameter, f θ ( · ) is a convolutional neural network (CNN) image encoder, Q θ ( · , · ) is a critic head, and [ · ] is a concatenation operator along a channel dimension. Subsequently, to learn the representations robust to visual perturbations, data augmentation to the visual observations is leveraged when training the encoder. Specifically, s t in Eq. (2) can be substituted with τ ( s t ) = ( τ ( o t -2 ) , τ ( o t -1 ) , τ ( o t )) , τ ∼ T , where τ ( · ) is a sampled transformation function from the transformation space T . However, we empirically found that these practices can cause the saliency imbalance between the stacked frames and easily fall into observational overfitting, resulting in performance degradation in unseen challenging environments.

Pitfall 1: Imbalanced Saliency Fig. 2(a) illustrates an example of the imbalanced saliency by SVEA [12] in the 'Cartpole' task. As described in Section 2, the attribution masks were acquired by thresholding the gradient value of the critic function with respect to input image frames [2]. In this example, an RL agent recognizes the regions of the salient objects in the two latest frames as having high saliency across all stacked frames. This makes the agent misidentify as salient pixels of o t -2 its background parts corresponding to the positions of the 'Cartpole' object in o t -1 and o t . As a result, o t -2 provides the agent with redundant information that is unnecessary for the task, leading to overfitting to training data [45, 5]. In a test environment with complicated backgrounds, the background information that is considered salient from o t -2 may act as noise that interferes with the decision-making process of the policy. As the latest frames are more closely related to the subsequent decision-making process, the agent tends to identify spatially consistent saliency maps based on these recent frames, leading to imbalanced saliency maps. We hypothesize that this phenomenon occurs because the encoder extracts features throughout concatenated images along the channel as in Eq. (2), forcing it to capture the same spatial saliency for all stacked images.

Pitfall 2: Observational Overfitting Observational overfitting [32] can arise when certain background elements move in synchronization with task-relevant objects. It is prone to occur particularly when the motion of the task object is not substantial, such as in the 'Cheetah, Run' task. Although data augmentation is leveraged during training, we found that observational overfitting can still occur. Furthermore, as the same augmentation is applied uniformly to all consecutive frames, it does not effectively prevent the agent from erroneously focusing on background elements. For instance, Fig. 2(b) shows an example of observational overfitting that occurs in the 'Cheetah, Run' task by SVEA, where the critic assigns more significant saliency to the ground than to the 'Cheetah' object. In a test

Training

SVEA

SimGRL

Test

SVEA

SimGRL

7-2

t-1

:

Training

SVEA

SimGRL

Test

SVEA

SimGRL

7-2

t-1

environment, if there is a lack of correlation between objects and backgrounds present in the training environment, the agent may struggle to make accurate decisions, thereby leading to challenges in generalization.

4 Method

To address the problems within the conventional practices used for the generalization of visual RL, we propose a Sim ple framework for G eneralization in visual RL (SimGRL) under dynamic scene perturbations.

4.1 Feature-Level Frame Stack

In Section 3, we identified the encoder structure that simultaneously encodes the stacked frames at the image level as the cause of the imbalanced saliency. We address this issue by introducing an architectural regularization strategy to slightly modify the encoder structure. This architectural modification involves encoding each frame individually and then encoding the stacked feature maps. To keep the computational cost almost unchanged, we partition the original encoder into two segments instead of adding new layers. Then, Eq. (2) can be modified as follows:

where f 1 θ ( · ) is an image encoder to encode individual frames, and f 2 θ ( · ) is a feature encoder for the stacked feature maps. While f 2 θ encodes both spatial and temporal structures from inputs, f 1 θ conducts only spatial encoding of each frame. This simple modification allows the critic to be implicitly trained to focus on the spatially salient pixels of individual frames, enabling the agent to distinctly identify the salient pixels of each frame. In our experiments, we will verify that this simple modification of the encoder is highly beneficial for generalization, especially in challenging test environments.

4.2 Shifted Random Overlay Augmentation

The random overlay augmentation [13] used in existing approaches augments input images through linear interpolation with a natural image ε ∈ R C × H × W randomly sampled from the Places [46] dataset that contains 1.8M diverse scenes:

•

•

•

(

𝑑𝑑ℎ

ℎ0

,

(

ℎ𝑘𝑘

,

⨁

Augmented State

𝜏𝜏

𝑆𝑆𝑆𝑆𝑆𝑆

(

𝑠𝑠

;

𝑙𝑙

,

𝜀𝜀

)

Shifted

Crop

be Cropped

𝛼𝛼

×

(1

-𝛼𝛼

) ×

(

,

𝑤𝑤1

)

State

𝑠𝑠

= (

𝑜𝑜1

,

𝑜𝑜2

,

𝑜𝑜3

)

𝑜𝑜1

𝑜𝑜2

𝑜𝑜3

Image Frames

Sample Natural

Image

𝜀𝜀

𝐻𝐻 �

=

𝐻𝐻

+2

𝑛𝑛 -

1

𝑙𝑙

�

𝑊𝑊

=

𝑊𝑊

+2

𝑛𝑛 -

1

𝑙𝑙

𝑊𝑊

𝐻𝐻

�

W

�

𝐻𝐻

~

Dataset

𝐷𝐷

Shifted Patches

𝑐𝑐

𝜀𝜀2

𝑐𝑐

𝑐𝑐

𝑤𝑤0

):

coordinate of the top-left corner of the center crop

𝑑𝑑𝑤𝑤 ~ 𝑈𝑈 -𝐿𝐿 , 𝐿𝐿 , 𝑤𝑤 𝑘𝑘 ) = ( ℎ0 + 𝑘𝑘 � 𝑑𝑑ℎ , 𝑤𝑤0 + 𝑘𝑘 � 𝑑𝑑𝑤𝑤 ) where s is a state that consists of a sequence of images ( o 1 , ..., o n ) , o i ∈ R C × H × W , D is a dataset and a common choice for the interpolation coefficient is α = 0 . 5 . The original random overlay augmentation uses the same natural image ε for all stacked frames. Contrarily, we introduce a shifted random overlay (SRO) augmentation to inject the task-irrelevant dynamics into the training images, which is depicted in Fig. 3. Considering a maximum shift length l for each shifting and a stacked frame number n , this method first samples a natural image ε ∈ R C × ̂ H × ̂ W from the dataset D , where ( ̂ H, ̂ W ) = ( H +2( n -1) l , W +2( n -1) l ) . Then, we crop n shifted patches ε c i ∈ R C × H × W from ε to augment each frame in a shifted way. Given an upper left corner coordinate ( h 1 , w 1 ) of H × W sized center crop at the center of ε , the coordinates ( h i , w i ) of the upper left corner of ε c i are selected as follows:

where dh, dw ∼ unif {-l, l } . Finally, using the cropped images ε c i from ( h i , w i ) in ε , the shifted random overlay augmentation is defined as:

where we adopt α = 0 . 5 . This augmentation provides two implicit advantages when training RL agents. Firstly, the inclusion of background elements in motion, independent of the task, enables the RL agent to perceive that rewards are solely associated with changes in the genuine task object. Consequently, the agent is trained to concentrate on the task object, disregarding the movement of background elements, thereby alleviating the issue of observational overfitting. Secondly, this augmentation method generates training data akin to real environments with dynamic backgrounds, enabling the agent to be robust in such conditions and enhancing generalization capability.

4.3 Simple Framework for Generalization in Visual RL (SimGRL)

Built on the SVEA [12] framework that leverages the data-mixing strategy between weak and strong data augmentations and computes the target Q-value using clean images for stability in SAC, we propose a Simple framework for Generalization in visual RL (SimGRL) under dynamic scene perturbations, which integrates the proposed two strategies. For the strong augmentation, we utilize the shifted random overlay instead of the original random overlay without shifting while the random shift [40] is employed as the weak augmentation. Considering that the clean images s t and s t +1 already have weak augmentation, i.e. , random shift, applied by default, the critic loss for SimGRL is defined as follows:

where q θ ( · , · ) is computed by Eq. (3), q tgt = r t + γq ˆ θ ( s t +1 , a ′ ) , a ′ ∼ π ϕ ( ·| f θ ( s t +1 )) , and ˆ θ is the parameter of the target network that is updated by the exponential moving average (EMA). The strong augmentation τ SRO ( · ; · , · ) is the shifted random overlay in Eq. (6). Consistent with the SVEA, we adopt the data-mixing coefficient β = 0 . 5 . When training the actor π ϕ , we leverage solely clean images like the existing methods [12, 13]. The overall framework of SimGRL is illustrated in Fig. 4 and the algorithm is summarized in Appendix C.

T中

:Data-mix

cat

:Concatenation

f

cat

Q

g

6

:Stop-grad

TargetEncoder

Target Critic

Target Q-value

St+1

f

cat

f

Qe

1 -β

Encoder

Critic

Q-value

TSRO(St;l,e)

4.4 Task-Identification (TID) Metrics

To quantitatively evaluate the capability to identify task-relevant objects in each stacked frame as salient, we introduce novel metrics, referred to as Task-Identification (TID) score and variance, based on the ρ -quantile attribution mask M ρ ( Q θ , s t , a t ) [2] described in Section 2.

TID Score TID score measures how much the model identifies the task object's pixels across stacked frames, which is defined as:

where N obj is the number of task object's pixels in input images, N M is the number of pixels in attribution masks M ρ , N obj M is the number of task object's pixels included in M ρ , and ρ is a quantile value for thresholding the attribution map. Note that all numbers consisting of the TID score are counted across the full consecutive frames. In Eq. (8), the first term quantifies the model's ability to identify the pixels of the task object, while the second term quantifies how accurately the model identifies the task object's pixels. These two terms trade-off depending on the size of the quantile value and are upper-bounded by 1, thus leading to an upper-bounded TID score by 1. With a model that perfectly identifies all task pixels, this upper-limit value can only be achieved along with an optimal ρ value, which is the quantile of the number of the task object's pixels in frames. This ρ value can be computed by ρ = 1 -N obj ( n × C × H × W ) , where n is the number of frame stack. We explain the impact of the ρ value on this TID score in Appendix F.2.

TID Variance TID variance measures how discriminatively the model distinguishes the task object's pixels in each frame, which is defined as:

each TID S i is upper-bounded by 1, the values of TID V ar become too small for meaningful comparison. Therefore, we multiply TID S i by 100 to obtain a variance with a more appropriate scale for comparison.

5 Experiments

In this section, we present the experimental results of SimGRL on the DMControl-GB [13] video benchmarks ('Video Easy' and 'Video Hard') at 500K simulated training frames to demonstrate the generalization capability under dynamic scene perturbations. To tackle the vision-based control tasks with continuous action space, we employ the SAC [8] as a backbone RL algorithm, and compare SimGRL against current state-of-the-art

Video Easy Level

Video Hard Level

Training

Test

Data Augmentation

⨁

methods for visual RL generalization including RAD [19], DrQ [40], SODA [13], SVEA [12], TLDA [42], SGQN [2], EAR [4], and CG2A [23]. For weak augmentation, RAD and SODA utilize random crop, while the others employ random shift, except for SAC, which does not use any data augmentations. In addition, all competitors leverage the original random overlay augmentation without shifting for strong augmentation except for RAD and DrQ, which use only weak augmentations. As default for SimGRL, we used the random shift augmentation [40] and denoted the images with only this weak augmentation as clean images with the data distribution of the training data. Additionally, we employed the proposed shifted random overlay (SRO) for strong augmentation in SimGRL and denoted images together with this augmentation as augmented ones. Implementation details are described in Appendix A, and further experimental results on other benchmarks of Distracting Control Suite [34] and robotic manipulation [14] are provided in Appendices B.7 and B.8, respectively.

5.1 Results on DMControl-GB

We evaluated the zero-shot generalization performance on 6 tasks in 'Video Easy' and 'Video Hard' benchmarks from DMControl-GB [13]. As illustrated in Fig. 5, the easy version shares certain structures with the training environment such as the ground and shadow, while the hard version shares nothing other than the agent's object by replacing all backgrounds with distracting videos. In Table 1, SimGRL demonstrates state-of-the-art performance in 4 out of 6 tasks in the Video Easy benchmark and achieves comparable performance in the remaining two tasks, including 'Walker, Walk' and 'Cheetah, Run'. On the other hand, our approach shows outstanding performance in all 6 tasks at the Video Hard level, where SimGRL achieves performance gain by 15 % on average compared to SGQN. Specifically, SimGRL outperforms existing methods by a significant margin in 'Cartpole, Swingup' and 'Cheetah, Run', which were previously difficult to solve. We suggest that the reason for this lies in the fact that 'Cartpole, Swingup' requires detailed identification of salient objects, while 'Cheetah, Run' needs to address the observational overfitting. Both issues are mitigated in SimGRL. We provide the training curves in Appendix B.6.

Computational Efficiency Our method has the advantage of improving generalization performance without any extra models or auxiliary losses. We compare SimGRL with SGQN [2], the existing state-of-the-art method in the Video Hard benchmark. In experiments, SimGRL can achieve a throughput of 9.54 FPS, which is 1.55 × efficient compared to SGQN's 6.16 FPS on a single NVIDIA TITAN RTX GPU. Fig. 6 shows the training curves for the zeroshot test performances over the wall-clock training time, where the averaged performances across the 6 tasks in Table 1 are presented. In this figure, SimGRL requires 44 % less wall-clock training time than SGQN to reach the same 500K frames for training, demonstrating the high computational efficiency of SimGRL. It is worth

noting that SimGRL has achieved considerable enhancements in both training efficiency and test

performance compared to the SOTA method, regardless of the typical trade-offs in terms of efficiency and performance.

Ablation Study To verify the effectiveness of the proposed regularizations, we compare the ablation variants with the SVEA baseline. Fig. 7 shows training curves for zero-shot test performances averaged across the 6 tasks of DMControl-GB at the Video Hard benchmark as a function of the number of stepped training frames. This result indicates that the additions of each regularization to SVEA can remarkably improve the generalization ability of the model. In particular, we emphasize that each of the two regularizations leads to sufficient improvement, where all variants of SimGRL achieve, on average, nearly 82 % better performance than SVEA. Full experimental results for each task are provided in Appendix B.1 and further in-depth discussions on the effectiveness of each regularization are described in Appendix B.2.

5.2 Analysis with TID Metrics

To analyze the two potential overfitting problems, namely imbalanced saliency and observational overfitting, we evaluate the TID metrics, including the TID score and variance. For clarity, we compare three methods of DrQ [40], SVEA [12], and SimGRL as the former serve as baselines for the latter. We analyze two representative tasks, namely 'Cartpole, Swingup' and 'Cheetah, Run', where each problem is prominently observed. Full evaluations for all algorithms and tasks are provided in Appendix F.3. The left plot in Fig. 8 depicts that the overall distribution of training scores of the TID metrics is divided into three regions. In this plot, both DrQ and SVEA exhibit results divided into two regions, either 'low score and low variance' or 'middle score and high variance', depending on the task, indicating that different phenomena appear in the two tasks. The 'low score and low variance' region can be interpreted as observational overfitting, which does not identify the correct object in all frames. On the other hand, the 'middle score and high variance' area suggests that the correct object is accurately recognized only in certain frames, implying imbalanced saliency. We note that although SVEA [12] incorporates a data-mixing strategy with strong augmentation to enhance the generalization capability of DrQ [40], both problems are still observed with only minor improvements over DrQ. In contrast, SimGRL reports comparatively high scores and low variances for both tasks, indicating that it can effectively identify task-relevant objects regardless of the type of task. The middle plot shows a positive correlation between training and test TID scores at Video Hard from DMControl-GB [13]. This suggests that a high discrimination ability during training can result in a high discrimination ability during testing. Finally, the right plot shows the generalization gaps, which are computed by the performance differences between training and testing, against the TID scores. This suggests that a high discrimination ability of the task object can lead to improved generalization capability.

'Variance'vs'Score'

'Test'vs'Train'(Score)

'Generalization'vs'Score'

350

0.6

800

(Train)

300

250

(Test)

0.5

0.4

Generalization Gap

600

200

Score(

0.3

400

150

0.2

100

200

0.1

50

X

0

0.0

X

0.2

0.4

0.6

0.0

0.0

0.2

0.4

0.6

0

0.0

0.2

0.4

0.6

TID Score (Train)

TID Score (Train)

TID Score (Train)

DrQ (Cartpole)

食

SVEA (Cartpole)

SimGRL (Cartpole)

x

DrQ (Cheetah)

x

SVEA(Cheetah)

x

SimGRL(Cheetah)

DMControl-GB(VideoHard)

800

mmw

Return

600

AverageR

400

Baseline (SVEA)

200

SimGRL-F(SVEA+Feat Stack)

SimGRL-S (SVEA+SRO)

SimGRL (SVEA+Both)

Q

O

1

2

3

4

5

Number ofFrames (x105)

6 Related Works

The field of domain generalization for RL has garnered considerable attention in recent years where various approaches aim to enhance the robustness of policies against visual changes. A promising approach is adapting the policy to a test domain. For example, PAD [11] suggests adopting a self-supervised task to obtain a free training signal during deployment. On the other hand, one method, proposed in [36], involves using randomly simulated RGB images. Similarly, [27] train domain-adaptive policies by randomizing dynamics during the training phase. Several works explore data augmentation techniques to improve policy generalization capacity [19, 40, 29]. For instance, RAD [19] and DrQ [40] achieve significant improvement through random crop and shift, while DrAC [29] automatically identifies the most effective augmentation with regularization terms for the policy and value function. Instead of relying solely on augmented data for policy learning, SODA [13] aims to decouple augmentation from policy learning by using soft-augmented data for policy learning and strong-augmented data for auxiliary representation learning. In a recent development, SVEA [12] designs a stabilized Q-value estimation framework to address instability issues under strong data augmentation in off-policy RL, while DBC [43] uses bisimulation metrics to learn a representation that disregards task-irrelevant information. VAI [39] extracts a universal visual foreground mask to provide an invariant observation to RL. Similarly, several methods leverage salient masks to encourage the agent to focus on import pixels. For example, TLDA [42] attempts to only augment the task-irrelevant pixels using masks obtained from the Lipschitz constant of the policy while SGQN [2] proposes saliency-guided self-supervised learning using a gradient-based saliency mask. EAR [4] attempts to learn environment-agnostic representations to enhance the robustness of policies against visual perturbations. On the other hand, several methods such as TIA [7], DRIBO [6], and RePo [47], leverage model-based RL through a recurrent state-space model (RSSM) [10, 9] to explicitly learn latent representations that focus on task-relevant features while discarding task-irrelevant ones. Our work focuses on improving the generalization ability of the RL agent based on implicit regularization approaches.

7 Conclusion

In this paper, we identified two key issues that degrade generalization in vision-based RL, particularly under dynamic scene perturbations, by employing a gradient-based attribution mask. To resolve these issues, we proposed a simple framework involving an architectural modification to the encoder and a new data augmentation. The proposed SimGRL algorithm has achieved state-of-the-art results on Video Hard environments in DMControl-GB, outperforming existing methods that have yet to address these challenges. Additionally, we introduced TID metrics to quantitatively evaluate the ability to discriminate task-relevant objects of RL agents. Using these metrics, we demonstrated that improving the identification of task-relevant objects can enhance the generalization capability of the vision-based RL agent.

Acknowledgments and Disclosure of Funding

This research was supported by the National Research Foundation of Korea (NRF) grant funded by the government of Korea (MSIP) (NRF2021R1A2C2006703), by the Basic Research Lab Program through the NRF of Korea (RS-2023-00222385) and by the Yonsei Signature Research Cluster Program of 2024 (2024-22-0161).

References

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

David Bertoin, Adil Zouitine, Mehdi Zouitine, and Emmanuel Rachelson. Look where you look! saliency-guided q-networks for generalization in visual reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV) , pages 839-847. IEEE, 2018.

Hyesong Choi, Hunsang Lee, Seongwon Jeong, and Dongbo Min. Environment agnostic representation for visual reinforcement learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 263-273, 2023.

Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. arXiv preprint arXiv:1511.06068 , 2015.

Jiameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi-view information bottleneck. In International Conference on Machine Learning , pages 6074-6102. PMLR, 2022.

Xiang Fu, Ge Yang, Pulkit Agrawal, and Tommi Jaakkola. Learning task informed abstractions. In International Conference on Machine Learning , pages 3480-3491. PMLR, 2021.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations , 2020.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning , pages 2555-2565. PMLR, 2019.

Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations , 2021.

Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in neural information processing systems , 34:3680-3693, 2021.

Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 13611-13617. IEEE, 2021.

Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal, Mohit Jain, and Xiaolong Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters , 7(2):3046-3053, 2022.

Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence , 101(1-2):99-134, 1998.

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning , pages 5639-5650. PMLR, 2020.

Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems , 33:19884-19895, 2020.

Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In International Conference on Learning Representations , 2020.

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334-1373, 2016.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Siao Liu, Zhaoyu Chen, Yang Liu, Yuzheng Wang, Dingkang Yang, Zhile Zhao, Ziqing Zhou, Xie Yi, Wei Li, Wenqiang Zhang, et al. Improving generalization in visual reinforcement learning via conflict-aware gradient agreement augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23436-23446, 2023.

Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. In International Conference on Learning Representations , 2017.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Karl Pearson. Vii. mathematical contributions to the theory of evolution.-iii. regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character , (187):253-318, 1896.

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3803-3810. IEEE, 2018.

Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542 , 2017.

Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 5402-5415. Curran Associates, Inc., 2021.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data , 6(1):1-48, 2019.

Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In International Conference on Learning Representations , 2020.

Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 , 2014.

Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite-a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722 , 2021.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 23-30. IEEE, 2017.

Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts , 6:100022, 2020.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Xudong Wang, Long Lian, and Stella X Yu. Unsupervised visual attention and invariance for reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6677-6687, 2021.

Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations , 2021.

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10674-10681, 2021.

Zhecheng Yuan, Guozheng Ma, Yao Mu, Bo Xia, Bo Yuan, Xueqian Wang, Ping Luo, and Huazhe Xu. Don't touch what matters: Task-aware lipschitz data augmentation for visual reinforcement learning. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 , pages 3702-3708. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.

Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations , 2021.

Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893 , 2018.

Zheng Zhao, Lei Wang, and Huan Liu. Efficient spectral feature selection with minimum redundancy. In Proceedings of the AAAI conference on artificial intelligence , volume 24, pages 673-678, 2010.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence , 40(6):1452-1464, 2017.

Chuning Zhu, Max Simchowitz, Siri Gadipudi, and Abhishek Gupta. Repo: Resilient modelbased reinforcement learning by regularizing posterior predictability. Advances in Neural Information Processing Systems , 36:32445-32467, 2023.

Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 3357-3364. IEEE, 2017.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

A Implementation Details

In this section, we describe the implementation details used for SimGRL. For a fair comparison, we followed the same architectural designs and hyperparameters used in [13], with only a minor difference in the encoder. For the feature-regularized encoder, we used 3 layers for the image encoder f 1 with 16 channels and 8 layers with 32 channels for the feature encoder f 2 , leading to the encoder with a total of 11 layers that are the same layer number as the encoder used in [13]. Subsequently, we projected the convolutional features of the last layer to 100-dimensional linear vectors, which are fed into actor and critic heads with 1024-dimensional hidden layers. We updated the actor parameters by freezing the encoder, and the target network parameters using an exponential moving average (EMA) with a rate of 0.01 for the critic head and 0.05 for the encoder. Both actor and target networks were updated every 2 critic updates. We used a minibatch size of 128, stacked 3 image frames each of size 3 × 84 × 84 as input for RL, and employed the Adam optimizer [17]. For the maximum shift length L in the shifted random overlay augmentation, we employed L = 6 , which leads to 3 × 108 × 108 size of a natural image ε before shifted cropping. In Fig. 9, we present examples of augmented images by the shifted random overlay. This figure exhibits that the proposed augmentation approach injects randomly moving backgrounds into the training images. All hyperparameters are summarized in Table. 2

𝑠𝑠

𝑡𝑡

𝜏𝜏

𝑆𝑆𝑆𝑆𝑆𝑆

(

𝑠𝑠

𝑡𝑡

;

𝜀𝜀

)

𝑠𝑠

𝑡𝑡

𝜏𝜏

𝑆𝑆𝑆𝑆𝑆𝑆

(

𝑠𝑠

𝑡𝑡

;

𝜀𝜀

)

丫Y

B Additional Experimental Results

B.1 Detailed Results on Ablation Study

We provide detailed results of the ablation study in Table 3. For DrQ, we utilized [ K = 1 , M = 1] [40]. In this table, by introducing weak augmentation using random shift into the vanilla SAC [8], DrQ demonstrates highly improved performance in image-based RL at the training level. While DrQ reports the best results in most tasks at the training level, the performance drastically decreases in testing levels, implying overfitting to the training environments. SVEA [12] introduces a datamixing strategy to enhance training stability under strong data augmentation, thereby improving DrQ's generalization capability in distracting backgrounds using random overlay [13]. However, the performance of SVEA still falls behind in challenging environments such as Video Hard. To address this issue, we proposed the feature-level frame stack and shifted random overlay augmentation, resulting in significant improvements compared to the SVEA baseline in Table 3, especially in the Video Hard environments. It is noteworthy that each of the proposed regularizations demonstrates significant improvements compared to SVEA, where SimGRL-F and SimGRL-S apply the featurelevel frame stack and the shifted random overlay augmentation to SVEA, respectively. Especially, all of SimGRL's variants reach the saturated performance in several tasks at the Video Hard level except for 'Walker, Walk' and 'Cheetah, Run'. For more complicated tasks such as 'Walker, Walk' and 'Cheetah, Run' tasks where performance saturation has not yet been reached, the integration of the regularizations ( i.e. , SimGRL) clearly shows the performance improvement. Fig. 10 shows the training curves of ablations for each task at Video Hard.

1000

Walker,Walk

1000

1000

Ball In Cup,Catch

Average Return

800

M

800

800

600

600

600

400

400

400

200

200

200

0

1

2

3

4

5

0

1

2

3

4

5

0

1

2

3

4

5

1000

Cartpole,Swingup

1000

Finger, Spin

400

Cheetah,Run

Average Return

800

800

300

600

600

200

400

400

200

200

100

0

1

2

3

4

5

0

0

1

2

3

4

5

0

1

2

3

4

Number of Frames (x105)

Number of Frames (x105)

Number ofFrames(x1o5)

Baseline

SimGRL-F

SimGRL-S

SimGRL

(SVEA)

(SVEA+Feat Stack)

(SVEA+SRO)

(SVEA+Both)

B.2 Discussion on Impacts of Proposed Regularizations

We discuss the impacts of the proposed regularizations based on the attribution masks and the TID metrics. First, we investigate the impacts in terms of the imbalanced saliency in the 'Cartpole, Swingup' task. In the left plot of Fig. 11, both SimGRL-F and SimGRL-S show increased TID scores compared to SVEA. However, they exhibit different effects on TID variance as SimGRLF significantly reduces it but SimGRL-S rather increases it. This indicates that the imbalanced saliency remains in SimGRL-S while SimGRL-F alleviates this issue. Therefore, the performance improvement of SimGRL-F can be interpreted as a mitigation of the imbalanced saliency problem, while the performance improvement of SimGRL-S is a bit ambiguous. For the reason of the result with SimGRL-S, we suggest that despite the persistence of imbalanced saliency, the explicit injection of dynamic background elements might have contributed to improved generalization in such conditions. Furthermore, even though it does not distinctly discriminate task objects between the stacked frames, the increased TID score of SimGRL-S can be attributed to its improved accuracy in identifying recognized objects in certain frames, as depicted in Fig. 12. This implies that the shifted overlay augmentation implicitly encourages the model to identify task objects more accurately. As a result, SimGRL, which integrates both architectural and data regularization, demonstrates that its agent can identify task-relevant objects across stacked frames with greater accuracy and discrimination.

In the right plot of Fig. 11, all of the SimGRL variants demonstrate remarkably increased TID scores in the 'Cheetah, Run' task. Since the 'Cheetah, Run' task has less movement of task objects than the 'Cartpole, Swingup' task, differences in TID variance values for this task are not effective. Therefore, the core issue in this task is the observational overfitting [32] to ground or shadow rather than imbalanced saliency between consecutive frames. The increase in the TID score implies an enhanced ability to identify task-relevant objects, which is demonstrated in Fig. 12. As expected, this shows that the application of the shifted random overlay augmentation encourages the agent to focus on the task object while disregarding background elements. On the other hand, it is noteworthy that the frame stack at the feature level also contributes to solving this problem. We suppose this is because encoding from a single image without temporal information allows the model to concentrate on each image's most salient visual features. Consequently, by integrating those strategies, the proposed approach, SimGRL, effectively addresses the two problems highlighted in this paper that hinder generalization in visual RL.

Cartpole,Swingup

Cheetah,Run

350

350

300

300

(Train)

(Train)

250

250

200

200

150

150

100

100

50

50

0

0

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

TID Score (Train)

TID Score (Train)

Baseline

SimGRL-F

SimGRL-S

SimGRL

(SVEA)

(SVEA+Feat Stack)

(SVEA+SRO)

(SVEA+Both)

SVEA

SimGRL-F

SimGRL-S

SimGRL

𝑡𝑡

𝑡𝑡-

1

𝑡𝑡-

2

Observation

𝑡𝑡

𝑡𝑡-

1

𝑡𝑡-

2

SVEA

SimGRL-F

SimGRL-S

SimGRL

Observation

6

B.3 Impact of Number of Layers in Image Encoder

18

Walker,Walk

12

10

Throughput (FPS)

8

6

4

2

0

01

3

6

6

Number of Layers in Image Encoder

Walker,Walk

800

600

TestReward

200

0

01

3

6

9

Number of Layers in Image Encoder

Walker,Walk

500

Generalization Gap

400

200

100

0

01

3

6

9

Number of Layers in Image Encoder

While maintaining the number of layers in the entire encoder at 11, we conducted the ablation study for the number of layers in the image encoder. To consider only the impact of the layer number, we leveraged the SimGRL-F model which employs the original random overlay without shifting for the strong augmentation rather than the shifted version. Depending on the number of layers in the image encoder, we investigated test rewards, generalization gaps, and throughputs in the 'Walker, Walk' task, where the Video Hard domain was employed for testing environments. The generalization gap was evaluated by the difference between the training and testing performances, and the throughput was computed by the FPS processed in the simulator on a single NVIDIA TITAN RTX GPU. The results presented in Fig. 13 demonstrate that the incorporation of the feature-level frame stack, facilitated by the image encoder, enhances the model's generalization capabilities in dynamic scenes. It is noteworthy that even if the number of image encoder layers is only 1, the generalization gap is remarkably reduced. Additionally, the generalization gap decreases when the number of layers in the image encoder is increased, indicating that incorporating feature maps with higher abstraction enables the improvement of generalization. However, despite the improvement in generalization performance, there does not seem to be a significant increase in practical test performance, implying a reduction in the training performance through the mitigation of overfitting. This can also be attributed to the diminished depth of the feature encoder, which is crucial for encoding temporal information. Moreover, increasing the layer number in the image encoder results in decreased throughput, primarily due to the increased computational burden associated with processing individual frames. Considering both generalization capability and computational efficiency, we selected 3 layers for the image encoder and 8 layers for the feature encoder for the experiments.

B.4 Different Data Augmentations

⨁

Augmented State

𝜏𝜏

𝐼𝐼-𝑆𝑆𝑆𝑆𝑆𝑆

(

𝑠𝑠

;

𝑙𝑙

,

𝜀𝜀

)

Shifted

Crop

Irregularly

Sample

Regions to be Cropped

𝛼𝛼

×

(1

-𝛼𝛼

) ×

(

,

𝑤𝑤1

)

State

𝑠𝑠

= (

𝑜𝑜1

,

𝑜𝑜2

,

𝑜𝑜3

)

𝑜𝑜1

𝑜𝑜2

𝑜𝑜3

Image Frames

Sample Natural

Image

𝜀𝜀

𝐻𝐻 �

=

𝐻𝐻

+2

𝑛𝑛 -

1

𝑙𝑙

�

𝑊𝑊

=

𝑊𝑊

+2

𝑛𝑛 -

1

𝑙𝑙

𝑊𝑊

𝐻𝐻

�

W

�

𝐻𝐻

~

Dataset

𝐷𝐷

Irregularly

Shifted Patches

𝑐𝑐

𝜀𝜀2

𝑐𝑐

𝑐𝑐

⨁

Augmented State

𝜏𝜏

𝐴𝐴-𝑆𝑆𝑆𝑆𝑆𝑆

(

𝑠𝑠

;

,

𝜀𝜀2

,

)

𝛼𝛼

×

(1

-𝛼𝛼

) ×

State

𝑠𝑠

= (

𝑜𝑜1

,

𝑜𝑜2

,

𝑜𝑜3

)

𝑜𝑜1

𝑜𝑜2

𝑜𝑜3

Image Frames

~

Dataset

𝐷𝐷

Sampled

Arbitrary

Natural Images

𝜀𝜀2

人

人

To investigate the impact of differently augmenting the stacked frames, we provide additional experiments under two types of additional augmentations: (1) randomly cropped patches from the same image and (2) completely different images. We denote the first method as irregularly shifted random overlay (I-SRO) and the second method as arbitrarily stacked random overlay (A-SRO). While the SRO augmentation crops patches in a regularly shifted manner as Eq. (5), I-SRO irregularly crops patches from a sampled natural image ε . In contrast to Eq. (5), the coordinates ( h i , w i ) of the upper left corner of each cropped patch become as follows:

where dh i , dw i ∼ unif {-l, l } . On the other hand, A-SRO samples different natural images ε i ∼ D with H × W sizes and augments each RL frame using the different images. The processes of I-SRO and A-SRO are illustrated in Fig. 14.

For clarity of the effect for each augmentation, we do not include the feature-level frame stack and compare them in the video hard level with SRO ( i.e. , SimGRL-S in the paper) that leverages regularly shifted cropped patches from the same image for data augmentation. Table 4 shows that augmentations using irregularly shifted patches (I-SRO) or completely different patches (A-SRO) across the consecutive frames can achieve comparable performance with SRO using the regularly shifted patches, implying that augmenting each frame in a different manner ( i.e. , employing SRO, I-SRO, and A-SRO) can achieve significant performance improvement over augmenting each frame uniformly ( i.e. , RO) for visual RL generalization under dynamic scenes. Among such augmentations, we have employed the (regularly) shifted random overlay ( i.e. , SRO) as our primary approach for data augmentation considering its superior averaged performance.

B.5 Adoption of SGQN

Based on its architectural simplicity and absence of additional auxiliary losses beyond the actor and critic losses, we adopted the SVEA [12] algorithm as our baseline to clarify our contribution. However, our approach that utilizes two types of regularizations can be seamlessly integrated with any other algorithms in a plug-and-play manner. On the zero-shot evaluation in the video hard test environments, we present the experimental results of SGQN [2] + F (Feature-level frame stack), SGQN + S (Shifted random overlay), and SGQN + S + F in Table 5. This table shows that the proposed regularizations can also contribute to improved generalization for other methods, such as SGQN, demonstrating the generality of our method.

B.6 Training Curves on DMControl-GB

In Figures 15, 16, and 17, we illustrate the training curves by re-implementing the comparison algorithms ourselves using available official codes over 5 seeds.

Walker,Walk

Ball In Cup,Catch

1000

1000

1000

Average Return

800

800

800

600

600

600

400

400

400

200

200

200

0

0

1

2

3

4

5

0

1

2

3

4

5

0

1

2

3

4

Cheetah,Run

Cartpole,Swingup

Finger,Spin

500

1000

1000

400

Average Return

800

800

600

600

300

400

400

200

200

200

100

0

0

1

2

3

4

5

0

1

2

4

5

0

1

2

3

4

Number of Frames (x105)

Number of Frames (x105)

Number of Frames (x105)

RAD

DrQ

SODA

SVEA

TLDA

SGQN

SimGRL (Ours)

Walker,Walk

Ball In Cup, Catch

1000

1000

1000

Average Return

800

800

800

600

600

600

M

400

400

400

200

200

200

+0

0

0

1

2

3

4

5

0

1

2

3

4

5

0

1

2

3

4

5

1000

Cartpole,Swingup

1000

Finger, Spin

400

Cheetah, Run

Average Return

800

800

300

600

600

200

400

400

100

200

200

0

0

1

2

3

4

5

0

0

1

23

4

5

0

0

1

23

4

NUmber of Frames (x105)

NUmberofFrames(x105)

NUmberofFrames(x1o5)

RAD

DrQ

SODA

SVEA

TLDA

SGQN

SimGRL (Ours)

Walker,Walk

BallInCup,Catch

1000

1000

1000

Average Return

800

800

800

600

600

600

400

400

400

200

200

200

0

0

1

2

3

4

5

0

1

2

3

4

5

0

1

2

3

4

5

1000

Cartpole, Swingup

1000

Finger, Spin

400

Cheetah,Run

Average Return

800

800

300

600

600

200

400

400

100

200

200

0

0

0

1

2

3

4

5

1

2

3

4

5

0

1

2

3

4

5

NumberofFrames(x1o5)

NumberofFrames(x1o5)

NumberofFrames(x1o5)

RAD

DrQ

SODA

SVEA

TLDA

SGQN

SimGRL(Ours)

B.7 Results on DistractingCS

We provide the additional benchmark results on the Distracting Control Suite (DistractingCS) [34], where camera pose, background, lighting, and colors continually vary throughout an episode. In Fig. 18, we depict some examples of the DistractingCS in intensity levels = [0 . 1 , 0 . 3 , 0 . 5] . In this experiment, we utilized the same hyperparameters used in DMControl-GB. Fig. 19 demonstrates SimGRL's significantly improved generalization performances in low intensities, compared to DrQ and SVEA. In particular, SimGRL is the only method that reported practical performances in the 'Ball in Cup, Catch' and 'Finger, Spin' tasks. Furthermore, SimGRL achieves additional performance gains in the 'Walker, Walk, Catch' and 'Walker, Stand' tasks. On the other hand, despite the still low performance in 'Cartpole, Swingup', there are clear signs of improvement. These results indicate the robustness of our proposed approach not only to dynamic background changes but also to various forms of distortions, including camera pose variations at moderate intensities. Finally, we note that generalization in extremely distorted environments, such as those with high intensities in the DistractingCS, remains an open problem and will be a subject of promising future research.

Walker,Walk

1000

1000

800

800

Average Return

AverageReturn

600

600

400

400

200

200

0

0

0.1

0.2

0.3

0.4

0.5

0.1

0.2

0.3

0.4

0.5

Intensity

Intensity

1000

Ball in Cup, Catch

600

Cartpole,Swingup

1000

Finger, Spin

500

Average Return

800

Average Return

400

Average Return

800

600

009

300

400

200

400

200

100

200

0

0

0

0.1

0.2

0.3

0.4

0.5

0.1

0.2

0.3

0.4

0.5

0.1

0.2

0.3

0.4

0.5

Intensity

Intensity

Intensity

DrQ

SVEA

SimGRL

B.8 Results on Robotic Manipulation

To verify the generalization capabilities of the proposed approach across various domains, we conducted additional experiments for robotic manipulation tasks [14], where the agents are trained for 250K steps. In these experiments, since the test environments introduce color and texture changes rather than structured distractions, SGQN [2] employed random convolution [20, 19] for strong augmentation. However, for SimGRL, we still utilized the shifted random overlay. In addition, for shifting, we employed two frames as an input rather than a single image setup as in SGQN. Fig. 20 exhibits examples of two robotic manipulation tasks, including 'Reach' and 'Peg in Box', and each of the testing environments. In these examples, SimGRL demonstrates its ability to appropriately capture salient regions, characterized by the robot arm and target positions. Table 6 reports the experimental results of the two tasks. Although this benchmark primarily features color and texture changes rather than dynamic background perturbations, SimGRL demonstrates superior results, achieving the best performance in 1 out of 3 tests in the 'Reach' task and in 2 out of 3 tests in the 'Peg in Box' task, and the second best in the remaining tests. It is noteworthy that SimGRL, which leverages only a variant of the random overlay, provides comparable performance to CG2A [23] that employs both random convolution and overlay augmentations.

Train

Test 1

Test 2

Test 3

Observation

𝑠𝑠 ⊙𝑀𝑀𝜌𝜌

(a) Reach

Observation

𝑠𝑠 ⊙𝑀𝑀𝜌𝜌

(b) Peg in Box

C Algorithm for SimGRL

Algorithm 1 A Sim ple Framework for G eneralization in visual RL (SimGRL).

1: Hyperparameters: Total number of training steps T , mini-batch size N , learning rate η , target network update rate λ , actor and target network update frequency M , transformation function set V for random shift, natural image dataset D for SRO, critic loss coefficient β , maximum shift length l in SRO, replay buffer B , discount factor γ .

ˆ .

D Analysis of Attentions

[Cartpole, Swingup]

[Cheetah,Run]

SVEA

SimGRL

SVEA

SimGRL

Image

1.0

0.8

Mask

0.6

0.4

Attention

0.2

0.0

t-2

t-1

t

t-2

t-1

t

t-2

t-1

t

t-2

t-1

t

In this paper, we demonstrated that it is important for the RL model not to be disturbed by taskirrelevant backgrounds for better policy generalization. To do this, we assumed that the RL agent should at least focus more on the task object than on the background, and employed the attribution masks to analyze whether the trained RL models correctly distinguish the task objects and backgrounds. Additionally, to quantitatively analyze how well RL models can distinguish between the task object and the background, we proposed the TID metrics.

In practice, RL policies may not make decisions based on the entire object. Instead, they could care about a few key points within such objects that are most predictive of the reward. To investigate whether the ability to distinguish between task objects and backgrounds also helps in identifying the key points that influence decisions, we provide attention maps obtained from the softmax of the attribution maps M ( Q θ , s t , a t ) = ∂Q θ ( s t ,a t ) ∂s t before thresholding for the binarized masks. Fig. 21 shows that the identification ability for the task objects of SimGRL also leads to focus on the most important key point parts such as the pole and body of the 'Cartpole' or legs of the 'Cheetah' in the attention maps. On the other hand, the SVEA baseline incorrectly considers the background features as key points.

E Comparison to Robust RL Algorithms

[Cartpole,Swingup]

DBC

TIA

DRIBO

SimGRL

Image

= = = == = = = = =

Mask

= = = = = = =

Masking

t-2

t-1

t

t-2

t-1

t

t-2

t-1

t

t-2

t-1

t

t-2

t-1

t

In response to the recommendation of the reviewers to investigate whether the pitfalls also apply to robust RL algorithms such as DBC [43], TIA [7], DRIBO [6], and RePo [47], we provide comparative experimental results for these methods. While our work focuses on addressing the generalization issue in a purely model-free vision-based RL setting, which requires the frame stack to encode temporal information, the robust RL approaches aim to explicitly learn robust representations against background distractions by utilizing model-based RL. To encode temporal information, these approaches, except for DBC, train transition dynamics by leveraging complex recurrent neural network (RNN) encoders which take only a single frame as input to the encoder, thus applying a concept similar to feature-level frame stacking. On the other hand, DBC uses a CNN encoder that takes stacked frames and trains a dynamics model that predicts the latent states of the next stacked frames. Fig. 22 shows that methods like TIA, DRIBO, and RePo, which use RNN-based encoders, do not struggle with the imbalanced saliency problem, successfully identifying each salient object across consecutive frames like SimGRL. In contrast, despite employing robust representation learning, DBC still struggles with the imbalanced saliency problem, showing a bias toward identifying pixels from the object in the most recent frame as salient across consecutive frames.

Cartpole,Swingup (Test)

800

DBC

TIA

Return

DRIBO

600

SVEA

Average

SimGRL

400

200

0

1

2

3

4

5

NumberofFrames(x1o5)

Fig. 23 shows the curves of zero-shot test performances for the 'Cartpole, Swingup' task in the video-hard environment according to the number of training frames. For a fair comparison, we applied the random overlay augmentation to the robust RL algorithms where following to [13], the data augmentation was applied only during the representation learning stage. As anticipated, robust RL algorithms like DRIBO and RePo, which can distinctly identify task-relevant objects, achieved better test performance (except for TIA) than methods that struggle with imbalanced saliency, such as SVEA and DBC. We suggest that the reason for the inferior performance of TIA is that the TIA agent struggles to train the Distractor Model, which explicitly captures the background, due to the more diverse background augmentations compared to the relatively limited video backgrounds from the Kinetics dataset [16] used in its original experimental setup. Notably, despite its straightforward structure and absence of additional representation learning stages, the proposed SimGRL achieved both sample efficiency and strong test performance, outperforming robust RL approaches like DRIBO and RePo that rely on model-based representation learning strategies.

F Details on TID Metrics

F.1 Dataset for TID Evaluation

To fairly compare TID metrics of comparison methods on the same images, we constructed a dataset for the evaluation in advance. As illustrated in Fig. 24, we excluded samples that were excessively static because the side effect of imbalanced saliency is diminished in such samples. Additionally, we excluded samples where the task object was not fully visible. Instead, we included dynamic samples to fully evaluate the impact of imbalanced saliency. To this end, as the samples obtained from random actions were less dynamic, we selected the images by running SVEA's policies, our baseline algorithm, up to 100K steps to obtain a variety of dynamic samples. Using those images, as depicted in Fig. 25, we constructed 60 pairs of images and ground truth (GT) masks of task objects for each training and testing environment in Video Hard. These GT masks are employed to count N obj and N obj M in Eq. (8). Here, N obj represents the pixel number in the GT mask while N obj M is determined by the pixel number in the overlapping regions between the GT and attribution masks.

𝑡𝑡-

𝑡𝑡-2

𝑡𝑡-1

𝑡𝑡

𝑡𝑡-2

𝑡𝑡-1

𝑡𝑡

𝑡𝑡-2

𝑡𝑡-1

𝑡𝑡

𝑡𝑡-2

𝑡𝑡-1

(a) Examples of Excluded Samples

2

𝑡𝑡-

1

𝑡𝑡

𝑡𝑡-

2

𝑡𝑡-

1

𝑡𝑡

𝑡𝑡-

2

𝑡𝑡-

1

𝑡𝑡

𝑡𝑡-

2

𝑡𝑡-

1

𝑡𝑡

(b) Examples of Included Samples

𝑡𝑡

Image

21

GT Mask

口

21

Image

GT Mask

F.2 Impact of Quantile Value

In Fig. 26, the impact of quantile ρ values for the ρ -quantile attribution mask [2] is illustrated. Additionally, the effect of the ρ value on the TID score can be explained as follows. When the value of ρ is too small, the region of a mask is enlarged, thus making it easier for task objects to be included in the attribution mask. This corresponds to the increased N obj M , leading to an increased N obj M N obj in the first component of the TID score in Eq. (8). Simultaneously, this also increases N M , balancing the TID score by decreasing N obj M N M that is the second component of the TID score. Conversely, a large value of ρ reduces the size of the mask N M as in Fig. 26, consequently reducing N obj M . On the other hand, the decreased N M might increase N obj M N M , balancing the TID score. Therefore, for the quantile value used to evaluate the TID score and variance, we used the optimal quantile value computed by ρ = 1 -N obj (3 × C × H × W ) rather than a hand-designed value. However, we used ρ = 0 . 95 in our qualitative illustrations of the attribution masks, as it empirically provided the most plausible visual representation of a masked region.

𝝆𝝆

=

𝟎𝟎

.

𝟗𝟗

𝝆𝝆

=

𝟎𝟎

.

𝟗𝟗𝟓𝟓

𝝆𝝆

=

𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐

𝑡𝑡

𝑡𝑡-

1

𝑡𝑡-

2

Observation

𝝆𝝆

=

𝟎𝟎

.

𝟗𝟗

𝝆𝝆

=

𝟎𝟎

.

𝟗𝟗𝟓𝟓

𝝆𝝆

=

𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐

𝑡𝑡

𝑡𝑡-

1

𝑡𝑡-

2

Observation

10

F.3 Full Results of TID Evaluation

3 We present comprehensive results of the TID evaluation, including a comparison of training TID variance, test TID score, and generalization performance against the training TID score. For the 'Walker' agent, we evaluated the TID scores on the 'Walk' task rather than 'Stand'. Fig. 27(a) reports higher TID scores and lower TID variances of our approach, indicating that SimGRL can distinctly identify the salient pixels across the stacked frames for all tasks. In particular, for the 'Ball In Cup' agent, measuring the TID scores accurately is difficult due to the relatively small size of the salient object. Nevertheless, SimGRL shows remarkably higher TID with lower variance scores than the competitors. These results demonstrate that SimGRL can effectively alleviate the two highlighted issues, 'imbalanced saliency' and 'observational overfitting'. In Fig. 27(b), we can observe that the high identification ability of the task object in the training environment can lead to a similar capability in test environments. Finally, Fig. 27(c) shows a positive correlation between generalization performance and the TID score, where smaller generalization gaps correspond to higher generalization performance. For the distributions in Figures 27(b) and 27(c), we present Pearson correlation coefficients [26] in Table 7. These results imply that the high identification

capability for task-relevant objects across stacked frames in input, as represented by the high TID score, can contribute to improving generalization performance.

(a) Plots of 'TID variance (Train) vs TID score (Train)'

(b) Plots of 'TID score (Test) vs TID score (Train)'

(c) Plots of 'Generalization gap vs TID score (Train)'

300

Walker, Walk

300

Ball In Cup,Catch

Cartpole,Swingup

300

Finger,Spin

300

Cheetah,Run

500

250

200

E

250

200

(ule

400

E

250

200

E

200

250

Variar

ance

150

C

150

300

ance

150

150

100

100

200

100

100

TID

50

50

100

50

50

0

0.0

0.2

0.4

0.6

0

0.0

0.2

0.4

0.6

0

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.8

0

0.0

0.2

0.4

0.6

TID Score (Train)

TID Score (Train)

TID Score (Train)

TID Score (Train)

TID Score (Train)

Walker, Walk

Ball In Cup, Catch

Cartpole,Swingup

Finger,Spin

Cheetah,Run

0.7

0.6

0.6

0.8

0.6

0.6

0.5

0.5

0.5

())

0.5

()

0.4

Score (Test)

0.4

(a)

0.6

()

0.4

Score

0.4

0.3

0.3

Score

0.4

0.3

0.3

aI1

0.2

0.2

aI1

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0.0

0.0

0.2

0.4

0.6

0.0

0.0

0.2

0.4

0.6

0.0+

0.0

0.2

0.4

0.6

0.0 +

0.0

0.2

0.4

0.6

0.8

0.0

0.0

0.2

0.4

0.6

TID Score (Train)

TID Score (Train)

TID Score (Train)

TID Score (Train)

TID Score (Train)

Walker, Walk

Ball In Cup,Catch

Cartpole,Swingup

Finger, Spin

Cheetah,Run

Generalization

Gap

800

600

Generalization

Gap

800

600

Generalization

Gap

700

600

500

Generalization Gap

800

600

Generalization

Gap

400

300

400

400

400

400

200

200

300

100

200

200

200

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

TID Score (Train)

TID Score (Train)

TID Score (Train)

TID Score (Train)

TID Score (Train)

RAD

DrQ

SODA

SVEA

TLDA

SGQN

SimGRL (Ours)

F.4 Further Examples of Attribution Masking

For the existing state-of-the-art visual RL approaches for generalization, we provide rich examples of the attribution masks and the corresponding masked observations, where we used the images employed in the TID evaluations. Additionally, we used the quantile ρ = 0 . 95 across all examples. Figures 28 and 29 depict the examples for training and Video Hard testing environments, respectively.

4

19

0

RAD

DrQ

SODA

SVEA

TLDA

SGQN

SimGRL

𝑡𝑡

𝑡𝑡-

1

𝑡𝑡-

2

번

Observation

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

益

7

1

번

번

8

0

19

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

U

%.

19

富

公

1

1

번

14

𝑡𝑡

𝑡𝑡-

1

𝑡𝑡-

2

9

번

RAD

DrQ

SODA

SVEA

TLDA

SGQN

SimGRL

Observation

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

1

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

1

儿

公

4

2

번

번

12

14

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

3

번

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

𝑡𝑡

𝑡𝑡-1

𝑡𝑡-2

0

3

번

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction include the claims made in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [No]

Justification: The paper has limitations, but those are not discussed in the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper includes experimental result reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper includes experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setting is presented in the paper in detail.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper includes the results that are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper includes experiments compute resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research was conducted with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no safeguards risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Creators or original owners of assets used in the paper, are properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SimGRL for vision-based DRL tasks. It tackles two challenges: the imbalanced saliency, where an agent repeatedly favors saliency maps from the recent one in stacked frames, and the issue of overfitting to particular background areas rather than concentrating on elements crucial to the task at hand. To overcome the first challenge, the proposed solution modifies the image encoding process, treating each frame separately before combining their features. To deal with the second issue of observational overfitting, this research uses an innovative method based on shifted random overlay augmentation. This method trains the agent to associate rewards with changes in the actual object of interest, thus learning to ignore background motions. Experiments were carried out using the DMControl-GB video benchmarks, evaluated by the TID score and variance metrics. The results show that SimGRL outperforms existing methods and the ablation study reveals each regularization technique independently contributes to notable enhancements.\n\nSTRENGTHS:\nIn this study, the focus is set on addressing critical obstacles in DRL and the author suggested both straightforward and impactful alteration to the model's structure as well as enhancement in the augmentation techniques. The empirical outcomes are reasonable and the statistical findings detailed in Table 1 provide persuasive evidence of the effectiveness of their method.\n\nWEAKNESSES:\nThe paper could benefit from an explicit articulation of its motivation and contributions. While multiple mentions are naturally made where the two challenges are being addressed, situating the motivation behind the work and a summary of contributions in a dedicated paragraph would enhance the strength of the paper. Although the devised SimGRL algorithm achieves top-tier performance on the Video Hard benchmark, the analysis of its performance on the Video Easy benchmark is not explored in depth, which presents an area for more detailed examination.\n\nQUESTIONS:\n1) The findings concerning Video Easy presented in Table 1 appear to be of lesser importance. Could you provide a more detailed explanation of these findings?\n\n2) Furthermore, Table 1 reveals that 'Cartpole, Swingup' and 'Cheetah, Run' exhibit noteworthy outcomes, although the significance is not observed across all tasks. While you have offered explanations for the strong performance in 'Cartpole, Swingup' and 'Cheetah, Run', can you clarify why there weren't substantial improvements in some other tasks? Does this indicate that the impact of your study might be constrained?\n\nLIMITATIONS:\nNo. The paper has limitations, but those are not discussed in the paper. The societal impact is not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two simple, yet crucial modifications to the Generalization in the Visual RL pipeline where during the test time, the visual observation consists of various degrees of dynamically varying backgrounds.\n\nConcretely, the authors identify two issues that inhibit RL agents from generalization: (i) imbalanced saliency and (ii) observational overfitting. To overcome these issues they propose two simple, yet effective solutions -- (a) Feature level frame stack where instead of stacking frames at the observation level, they stack at a feature level after obtaining spatial features of the images and (b) Shifted Random Overlay Augmentation where instead of applying data augmentation to the stack of frames, a shifted version of the data augmentation is applied to simulate the dynamically moving background effect which in turn helps learn more generalizable policy.\n\nThrough their experiments on the DM-Control (and robotic manipulation in the appendix) with varying backgrounds, the authors show the generalization capability of their proposed SimGRL (Simple framework for Generalization in visual RL) framework.  \n\nAdditionally, the authors also propose Task-Identification (TID) Metrics to evaluate how much of the policy can capture the task-relevant objects in each stacked frame.\n\nSTRENGTHS:\n1. A key strength of this paper is the simplicity of the proposed solutions to address the generalization issue and these simple changes are quite effective as shown in the experiments.\n\n2. The intuition behind both the proposed solutions is clear and straightforward.\n\nWEAKNESSES:\n1. I do find myself in confusion with respect to the Attribution masking argument. I believe that the proposed solutions of Feature-level frame stack and Shifted Random overlay augmentation are both valid solutions to a generalizable policy irrespective of the authors inclusion of Attribution masking. My specific concern is as follows: \n\n    What is the guarantee that a clear thresholded gradient masking of the critic wrt the input, which consists of the complete segmentation of the relevant objects and agent in the mask will lead to a better generalizable policy? It can be possible that a generalizable policy would care about only a few key points that are predictive of the reward -- and hence be able to accurately predict the Value of a state, and in that sense, an accurate segmentation of the attribution mask need not be necessary. To give an example, in the robotic manipulation task of reach -- the reward function is typically shaped as some function of the Euclidean distance between the robot's end-effector and the goal position. In such a situation, the Value function shouldn't really be bothered about where the other parts of the robot are.\n\n2. The weakness in (1) ties into my second concern where if only a few parts of the agent are responsible for achieving (and predicting) reward, then I don't see the value of the proposed TID metric. This assumes that my entire object and agent in the scene is responsible for the policy, which as argued in (1) is not the case in most scenarios.\n\n3. [Relatively minor concern, not taken into account for the final scoring] It would be beneficial to the reader if the proposed components in Figure 4 could be highlighted with say a different color and a distinction between the existing pipeline of SVEA be made.\n\n4. On the experimentation end, I feel that the evaluation is very limited to DM-control tasks. While SimGRL performs quite well on Video-Easy and Video-Hard settings of DM-Control, and 2 other tasks of Robotic manipulation, I would have liked to see a more comprehensive evaluation of various visual RL environments. Specifically, I'd recommend performing evaluation on RL-ViGen [1] (https://gemcollector.github.io/RL-ViGen/)  which consists of a collection of diverse visual RL tasks. This would further strengthen this work's results.\n\n5. On L263, where authors say: \n> SimGRL demonstrates state-of-the-art performance in 4 out of 6 tasks in the Video Easy benchmark\n\n    One has to be careful while making such claims especially when the difference on one such task is +1 -- which in DM-Control doesn't really mean anything. I would ask the authors to look at IQM and Probability of Improvement metrics to compare SimGRL to other methods from [2] which has shown to be more reliable than taking an average over seeds. This does not involve re-running the experiments, the existing checkpoints and returns can be directly used over their wrapper.\n\nQUESTIONS:\n6.  Observational Overfitting: For this, one of the core arguments that authors provide is that because the same augmentation is applied uniformly to all frames the agent can tend to still focus on the background. I'm wondering how a baseline for this where you have different data augmentations for every frame in the history of stacked frames(say jittering, slight rotation, blur, overlay, random convolution, random shift) performs?\n\n7. In the Robotic manipulation results what does Test {1, 2, 3} mean? Does it mean running an evaluation on a _single episode_ for 5 seeds?\n\n----\n**References**\n\n[1] RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization, Zhecheng Yuan et al., NeurIPS 2023.\t\n\n[2] Deep Reinforcement Learning at the Edge of the Statistical Precipice, Rishabh Agarwal et al., NeurIPS 2021.\n\n----\n**Reason for current rating**: Because of my two major concerns around Attribute masking (and TID metrics), as well as lack of extensive experimentation on diverse RL environments, I am leaning towards a weak reject. However, my final decision would be based on authors' rebuttal, other reviewers comments and the discussion with authors during the rebuttal period. I'd encourage the authors to ask me any clarification questions for any of the experiments if they have any during the rebuttal/discussion period.\n\n-----\n\n**Post rebuttal update** -- As the authors have addressed all my major concerns, I have decided to increase my score to **Weak Accept**.\n\nLIMITATIONS:\nThe authors don't mention any limitations of their work. It would be good to have a couple of lines highlighting where their method may fail. For instance, potentially large changes in background can lead to failure or their Feature stacking would work only in situations where Frame stacking is used -- model-based works like Dreamer do not use any frame stacking because of a recurrent network (such as GRU) that handles the history of states.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper first identify two pitfalls with the visual RL via gradient-based attribution mask, i.e., imbalanced saliency and observational overfitting. To address these two pitfalls, the paper proposes two novel modifications. One is for the encoder where encode each frame and stack encoded representations instead of encoding stacked frames directly. The other is a newly proposed shifted  randomly overlay augmentation method to address the observational overfitting.\n\nSTRENGTHS:\n- By and large, the paper is written well and presented clearly.\n- The section on pitfalls within visual RL algorithms is also interesting, and highlights the authors attention to the RL robustness and generalization problems.\n\nWEAKNESSES:\n- One concern I have about the paper is the first pitfall where the imbalanced saliency is specifically applied to visual RL algorithms that apply stacked frames. However, many robust RL algorithms, e.g., DBC, SLAC, TiA, DRIBO, RePo, take only single frame as the input to the encoder which apply similar ideas as the feature-level frame stack. These algorithms also explicitly take previous frame stack as input with an encoder $p_\\theta(s_t | s_{t-1}, a_{t-1}, o_t)$. It would be interesting to investigate whether these methods also suffer from the imbalanced saliency and compare the proposed feature-level stack method with them in terms of performance and efficiency.\n- Though the paper identifies the pitfalls with attribution methods, it would also be interesting to see attribution masks of SimGRL in different settings, i.e., clean, video easy and video hard. This would help validate that the identified pitfalls are sufficiently addressed by SimGRL.\n- Fig 5 and demos showed along the paper may be misleading that the policy is purely learned from the clean environment without any background perturbation. However, the proposed randomly shifted overlay augmentation explicitly introduces background perturbations into the training process.\n\nQUESTIONS:\n- Could you comment on whether the pitfalls also apply to robust RL algorithms like DBC, SLAC, TiA, DRIBO and RePo?\n- Could you comment on how different attribution methods may affect the identified pitfalls and TID metrics?\n\nLIMITATIONS:\nYes, the author addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SimGRL, a framework aimed at enhancing generalization in vision-based deep reinforcement learning (RL) under dynamic scene perturbations. It addresses two critical issues in existing visual RL methods: imbalanced saliency and observational overfitting. SimGRL employs architectural modifications to the image encoder and a novel shifted random overlay augmentation technique. Extensive experiments demonstrate SimGRL's superior generalization capabilities, achieving state-of-the-art performance on the DeepMind Control Suite and other benchmarks.\n\nSTRENGTHS:\nInnovative Architectural Modification: The proposed modification to stack frames at the feature level instead of the image level effectively addresses the imbalanced saliency issue, ensuring the agent focuses on spatially salient features in each frame .\nNovel Data Augmentation Technique: The shifted random overlay augmentation introduces dynamic background elements during training, which helps the agent to focus on task-relevant features and ignore irrelevant background changes .\nComprehensive Evaluation: The method is thoroughly tested on multiple benchmarks, including the DeepMind Control Suite and DMControl-GB, demonstrating superior generalization performance compared to state-of-the-art methods.\nQuantitative Metrics: The introduction of TID metrics provides a quantitative way to evaluate and understand the method's effectiveness in identifying task-relevant features, correlating high TID scores with improved generalization performance.\nComputational Efficiency: The method achieves significant improvements without requiring additional auxiliary losses or networks, maintaining computational efficiency.\n\nWEAKNESSES:\nDependency on Augmentation Quality: The effectiveness of the shifted random overlay augmentation relies on the quality and diversity of the natural images used for augmentation, which may limit its performance in certain real-world scenarios where such images are not available or appropriate.\nAd Hoc approach: the authors offer specific fixes to observed shortcomings in vision-based deep RL in simulated environments, and it remains to be seen if these fixes are an hand-crafted overfitting to specific tasks that won't generalize to other tasks.\nFinally, while the method shows impressive results in simulated environments, its applicability and effectiveness in more realistic real-world scenarios have not been thoroughly validated.\n\nQUESTIONS:\nHow does the diversity of natural images used in the shifted random overlay augmentation impact the generalization performance of the RL agents? Are there specific types of images that are more effective?\nAre there any plans to test SimGRL in real-world environments, and what challenges are anticipated in such scenarios? How does the method handle real-world dynamic perturbations that are not present in simulated benchmarks?\nWhat are the effects of using different numbers of frames in the feature-level frame stack on the performance and generalization capability of SimGRL?\nHow sensitive is the performance of SimGRL to the choice of hyperparameters such as the number of layers in the image encoder and the maximum shift length in the shifted random overlay augmentation?\n\nLIMITATIONS:\nThe architectural modifications and data augmentations, while effective, still introduce some computational overhead, which may impact training times and computational resources required.\nThe method is specifically designed for vision-based RL and may not directly apply to other types of RL tasks without visual components.\nThe shifted random overlay augmentation relies on the quality and diversity of natural images used for augmentation, which may limit its performance in scenarios where such images are not available or appropriate.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}