A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations
Authors: Wonil Song
Hyesong Choi
Kwanghoon Sohn
Dongbo Min
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual Reinforcement Learning
Generalization
Primary area: reinforcement_learning

Abstract: In the rapidly evolving domain of vision-based deep reinforcement learning (RL), a pivotal challenge is to achieve generalization capability to dynamic environmental changes reflected in visual observations.
Our work delves into the intricacies of this problem, identifying two key issues that appear in previous approaches for visual RL generalization: (i) imbalanced saliency and (ii) observational overfitting.
Imbalanced saliency is a phenomenon where an RL agent disproportionately identifies salient features across consecutive frames in a frame stack. 
Observational overfitting occurs when the agent focuses on certain background regions rather than task-relevant objects.
To address these challenges, we present a simple yet effective framework for generalization in visual RL (SimGRL) under dynamic scene perturbations.
First, to mitigate the imbalanced saliency problem, we introduce an architectural modification to the image encoder to stack frames at the feature level rather than the image level.
Simultaneously, to alleviate the observational overfitting problem, we propose a novel technique called shifted random overlay augmentation, which is specifically designed to learn robust representations capable of effectively handling dynamic visual scenes.
Extensive experiments demonstrate the superior generalization capability of SimGRL, achieving state-of-the-art performance in benchmarks including the DeepMind Control Suite.

A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations
Wonil Song Yonsei University Seoul, South Korea
Hyesong Choi Ewha Womans University
Kwanghoon Sohn Yonsei University Seoul, South Korea
Seoul, South Korea hyesongchoi 20wpnnchhoii000000 2000000110111m111 com
swoni192@yonsei.. ac kr
khsoon@yonseei. ac kr
Dongho Min Ewha Womans University Seoul, South Korea dbmin@ewha.c ac.kr kr
Abstract
In the rapidly evolving domain of vision-based deep reinforcement learning (RL), a pivotal challenge achieve generalization capability dynamic environmental changes reflected in visual observations. Our work delves into the intricacies of this problem, identifying two key issues that appear in previous approaches for visual RL generalization: (i) imbalanced saliency and (ii) observational overfitting. Imbalanced saliency is phenomenon where RL agent disproportionately iden- tifies salient features across consecutive frames in frame stack. Observational overfitting occurs when the agent focuses on certain background regions rather than task-relevant objects. To address these challenges, we present simple yet ef- fective framework for generalization in visual RL (SimGRL) under dynamic scene perturbations. First, mitigate the imbalanced saliency problem, an architectural modification to the image encoder to stack frames at the feature level rather than the image level. Simultaneously, to alleviate the observational overfitting problem, we propose a novel technique called shifted random overlay augmentation, which is specifically designed to learn robust representations capable effectively handling dynamic visual scenes. Extensive experiments demonstrate the superior generalization capability of SimGRL, achieving state-of-the-arr perfor- mance benchmarks including the DeepMind Control Suite.
introduce
Introduction
Deep reinforcement learning (RL) utilizing visual observations has achieved remarkable success across diverse domains, including robotic manipulation [21], video games 225 38) and autonomous navigation 24 48). However, acquiring generalizable RL policies across diverse environments remains challenging, mainly due overfitting 44] in the high-dimensional observation space [32)
To obtain robust policies invariant to visual perturbations, variety of approaches based on domain randomization [36] 28 and data augmentation [31 19 29 B have been widely proposed. These approaches operate under the assumption that exposing an agent to various augmentations during the training phase enhances its adaptability to unseen domains. Despite the encouraging results, performance still falls behind in challenging environments with dynamically changing backgrounds.
"Corresponding Author
Webssite and code are available at: http:://o--soglll.ttttu.ooo//
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

DMConnrol-Co GB Video Benchmarks
Stacked Frames
Stacked Frames
3mdRL
t-2
t
t-2 2
002A
550W
TLDA
SYEA
Observation
500A
u3s0 ian
150
200
300
Average Score
.000
500
850
700
D0C
SOM,
(a) DMControl Generalization Performance
(h) Imbalanced Saliency
(c) Observational Overfitiing
Figure 1: (a) Average performances 6 tasks in DMControl-GB, In contrast to other methods with significant performance degradation in Video Hard, our proposed SimGRL demonstrates robust performance across all benchmarks. (b)-(c) Examples of two problematic phenomena that can cause overfitting in visual RL generalization. The background structures in the red boxes are correlated with the movement of the task object. and M, represent the stacked frames and attribution masks, respectively. Attribution masks this figure were obtained using the critic trained by DrQ [40].
Fig. ((a) shows the notable performance degradation of existing approaches when comparing 'Video Hard scenes with dynamically changing backgrounds to less dynamic 'Video Easy" scenes in the DeepMind Control Quite-Generalizatio Benchmark (DMControl-Gii
13
Using a gradient- attribution mask Mp 2] we first investigate the causes of the degradation in generalization in such challenging environments by examining salient regions across consecutive stacked frames used as an RL input. Based on our analysis, we empirically identified two phenomena, highlighting them key causes of performance degradation: (i) what we refer as imbalanced saliency and (ii) observational overfitting [32]. In the DMControl 33 37]. Fig. [(b) depicts an example of the imbalanced saliency in 'Cartpole, Swingup' task, where salient features across all stacked frames are biased to the regions corresponding to the task objects in the latest two frames. Fig. shows an example the observational overfitting that occurs in 'Chectah, Run' task. In this case, the RL agent misidentifies the ground as more salient than the 'Cheetah' object. These two problems contribute overfitting to the training environment, making generalization of the RL agent even more challenging.
In image-based RL, the effectiveness of regularization has been demonstrated in numerous studies. DrQ [40] and RAD [19), each employing random shift and random crop augmentations as data regularization on input images, achieved remarkable performance improvements over vanilla SAC 8. Furthermore, (32) exhibited that regularization effects from architectural modifications such as overparameetrrizatoo residual connections can help avoid observational overfitting and reduce the generalization gap.
In this context of research, we introduce a Simple yet effective framework for Generalization in visual RL (SimGRL) under dynamic scene perturbations. Firstly, we empirically found the image-level frame stack, commonly used in traditional vision-based model-free RL approaches [25 18 41 19), to be the main factor causing the imbalanced saliency problem. To address this issue, we propose a simple architectural modification of image encoder, which employs feature-level frame stack instead of the image level. This involves extracting individual feature maps for each frame from the shallow layers of the encoder, stacking them along channels, and encoding the stacked feature maps through the remaining layers. Since this approach considers solely individual frames during the initial encoding, the agent is trained focus on spatially salient features in each consecutive frame that are essential for a given task, alleviating the imbalanced saliency problem of Fig. [[b)) Secondly, to address the observational overfitting problem by encouraging the agent focus on the task object rather than backgrounds, we propose new data augmentation called shifted random overlay, which a modified version random overlay augmentation [3]. This augmentation directly injects background dynamics irrelevant to the given task into the agent during the training phase by interpolating natural images moving a random direction on each frame. By enabling the agent to implicitly learn to ignore task-irrelevant backgrounds and focus on task-relevant pixels, this approach alleviates the observational overfitting problem of Fig. [Tc) Furthermore, this augmentation enables the agent to adapt to test environments with dynamic perturbations in the surrounding backgrounds.

---

We verify that these strategies can remarkably improve generalization in challenging environments, achieving state-of-the-a performance without using any additional auxiliary losses or networks.
Furthermore, utilizing the attribution mask Mp (2, we introduce novel metrics, called Task- Identification (TID) metrics, consisting of TTD score and TID variance, to quantitatively evaluate the discrimination ability on salient regions. With the proposed metrics, we quantitatively analyze the two problematic phenomena in the existing approaches. demonstrating the excellent discrimination ability of SimGRL. Moreover, we show a tendency of positive correlation between the TID score and RL generalization performance, emphasizing the importance of accurately identifying salient features in input images.
Our contributions include the following aspects:
By utilizing gradient- based attribution masks, we highlight the two core issues of imbalanced saliency and observational overfitting, which hinder the generalization of visual RL for most model-free RL settings. Additionally, we propose TID metries to measure the discrimination ability of an RL agent on task objects, providing insights into these issues.
To address these problems, we propose architectural and data regularization methods through a modification to an encoder structure and an introduction of new data augmentation.
We achieve state-of-the-art performances across video benchmarks of DMControl-GB 13 DistractingCS [34], and robotic manipulation tasks [4]
2 Background
Visual RL and Generalization We consider partially observable Markov decision process (POMDP) problem M OS, r,,,), where S is state space, O is observation space, A is action space, P : x A S is the transition function that defines the conditional probability distribution ((s st, ae) over next states given state &t E and an action a E taken time f, S x A - IR is a reward function, and y E [0, 1) is discount factor. In the POMDP problem, such as the visual RL, because only the high-dimensional observations o, O can be observable [55
state is defined as a sequence of the & consecutive image frames 8; (Ot-k+ k-+ I, .... 1 0z) [25]
Without loss of generality, we will set = for the sake of notational simplicity. RL aims to learn a policy π. : S - A that maximizes the expected sum of discounted rewards E, |Ei= t 7"r(s,, r+)] In
this work, we focus the generalization problem POMDPs M (S, Ô A, r, 7), where the states E S are constructed from the perturbed observations E Ô, and POMDP M sampled from the space of POMDPs M, M ~ M.
Deep Q-Learning and Soft Actor-Critic Deep Q-learning 25 is common model-free RL algorithm that aims to learn parameterized state-action value function Qa(st, (t), where 0 is the deep neural network parameters of the Q-function and action is greedily selected as the one with the maximum value a argmax, Qo (s:, a) at time t. The training of the Q-function is achieved by minimizing a mean squared error of the Bellman residuals:
E E 1) 5l(
15 (Qo (s a.) (r. + max 029a ( (s:41,
(1)
where å is the parameters of the target Q network Qtat and
replay buffer. To increase the stability, the parameter of the target Q network is slowly updated by the exponential moving average (EMA) 0 = X0 + (1 λ) with À & 1 [22). For the continuous action space, rather than the greedy sampling, a parameterized actor ()())) is employed as the policy, where Ø is the neural network parameters of the actor. Soft Actor-Critte (SAC) (8] is common actor-critic algorithm with state-action value Qs(st, at) and a stochastic policy Tss (a))))))). and temperature parameter a, which aims to optimize 7-discounted maximum-entropy objective [49].
In visual RL, a parameterized encoder fo RCCH*W dimensional image inputs and shared by both the actor and critic, where d is dimension of the encoded feature. Consistent with the prior visual RL approaches [8 the encoder with the critic and freeze it during actor updates, where the encoder learns representations for RI. tasks by the critic loss. For notational simplicity, we denote all parameters updated by the critic loss as 0 and the actor parameters ø.
Rd is employed to compress the high-
40 [2]
we jointly train
3

---

Gradient-based Attribution Mask An attribution map. also known as saliency map, is designed to visualize the salient pixels in imput images for given tasks. common approach to computing the attribution map is wradient-based method [30 3 which indicates how sensitive the task
prediction is to perturbations in the input pixels. Consistent with SGQN (2). we employ guided
backpropagation 333 to compute the attribution map M (Q0, St, az) = [Qu(s..a.) where Sn and
ds:
(Q0, St, az) E RC Then, the binarized attribution mask Mo(Qo, St, at), referred to as p-quantile attribution mask, is computed by thresholding M (Qp, St, az) by the p-quantile, where Mo(Qo, 8t, (t.,..)))) = if the pixel value of M (Qu, St, at) (s.j..) belongs to the p-quantile of highest values for M (Qe, st, at), ), and 0 otherwise. We use this attribution mask to investigate the cause of the degradation of the generalization performance of existing methods in challenging environments such as Video Hard [[3] Furthermore, to justify our analysis, we will introduce new metries that quantitatively measure the ability to identify salient pixels in Section [.44 Note that we leverage the attribution mask only for analysis without involving it in the training process.
3 Pitfalls within Conventional Practices in Visual RL Generalization
The generalization ability of RL agents is often degenerated in challenging environments characterized by dynamic perturbations and different structures compared to the training environment. In this section, we investigate the causes of the overfitting to the training environment observed in existing visual RL methods on the DMControl Generalization Benchmark (DMControl 13
Conventional practices used to train the visual RL agent for generalization I 2
42 4
include
frame stack and data augmentation. First, to reflect the temporal structure of the input state, Q-value
is predicted using stacked frames [25 18 41
40
19 instead of single image as follows:
qa (s dz) -
or),,
8: = (o: 0c 0z),
where is a neural network parameter, fo(.) is a convolutional neural network (CNN) image encoder, -)(:--) is critic head, and is a concatenation operator along channel dimension. Subsequently, to learn the representations robust to visual perturbations, data augmentation to the visual observations is leveraged when training the encoder. Specifically, St in Eq. 2 can be T ~ T, where r(-) is sampled transformation function from the transformation space T. However, we empirically found that these practices can cause the saliency imbalance between the stacked frames and easily fall into observational overfitting, resulting in performance degradation in unseen challenging environments.
substituted with r(x;) = (T(o:-2), 7 (0.
Pitfall 1: Imbalanced Saliency Fig. 2(a) illustrates an example of the imbalanced saliency by SVEA 112 in the 'Cartpole' task. As described in Section 2 the attribution masks were acquired by thresholding the gradient value of the critic function with respect to input image frames (2] In this example, an RL agent recognizes the regions of the salient objects in the two latest frames as having high saliency across all stacked frames. This makes the agent misidentify as salient pixels of 0r-2 its background parts corresponding to the positions of the 'Cartpole' object in 0;-1 and or. As result, o,-2 provides the agent with redundant information that is unnecessary for the task, leading to overfitting to training data [45] []. In a test environment with complicated backgrounds, the background information that is considered salient from Ot-2 may act as noise that interferes with the decision- makinng process of the policy. As the latest frames are more closely related to the subsequent docision-making process, the agent tends to identify spatially consistent saliency maps based these recent frames, leading to imbalanced saliency maps. We hypothesize that this phenomenon occurs because the encoder extracts features throughout concatenated images along the channel as in Eq. (2). forcing it to capture the same spatial saliency for all stacked images.
Pitfall 2: Observational Overfitting Observational overfitting 32 can arise when certain back- ground elements move in synchronization with tesk-relevant objects. is prone to occur particularly when the motion of the task object is not substantial, such as in the 'Cheetah, Run' task. Although data augmentation is leveraged during training, we found that observational overfitting can still occur. Furthermore, as the same augmentation is applied uniformly to all consecutive frames, it does not effectively prevent the agent from erroneously focusing on background elements. For instance, Fig. shows an example of observational overfitting that occurs in the 'Cheetah, Run" task by SVEA, where the critic assigns more significant saliency to the ground than to the 'Cheetah' object. In test

---

Stecked Frames
18
(a) Attribution masking examples in 'Cartpole, Swingup"
Training
SVIA
SimGRi
Test
IVZA
SimGRL
t-2
Stacked Frains
(b) Attribution masking examples in 'Cheetah, Run'
Figure 2: Examples of attribution masks and masked frames. Compared to SVEA that falls into the imbalanced saliency and observation overfitting in the 'Cartpole, Swingup' and 'Cheetah, Run' tasks, respectively, the proposed SimGRL accurately identifies the true salient pixels even in challenging 'Video Hard' test environments of DMControl-Git We provide further examples of the attribution masks and masked salient regions for various environments and algorithms in Appendix F.4
environment, if there is lack of correlation between objects and backgrounds present in the training environment, the agent may struggle to make accurate decisions, thereby leading to challenges in generalization.
4 Method
To address the problems within the conventional practices used for the generalization of visual RL, propose Simple framework for Generalization in visual RL (SimGRL) under dynamic scene perturbations.
4.1 Feature-Level Frame Stack
In Section we identified the encoder structure that simultaneously encodes the stacked frames the image level the cause of the imbalanced saliency. We address this issue by introducing an architectural regularization strategy to slightly modify the encoder structure. This architectural modification involves encoding each frame individually and then encoding the stacked feature maps. To keep the computational cost almost unchanged, we partition the original encoder into two segments instead of adding new layers. Then, Eq. (2) can be modified as follows:
qe (st ( (or-2), (oL-1), (o:))), (o a.),
(01-2, 0z):
(3)
where (.) is an image encoder encode individual frames, and (-) is feature encoder for the stacked feature maps. While f encodes both spatial and temporal structures from inputs. f conducts only spatial encoding each frame. This simple modification allows the critic to implicitly trained to focus the spatially salient pixels of individual frames, enabling the agent distinetly identify the salient pixels of each frame. In our experiments, we will verify that this simple modification of the encoder is highly beneficial for generalization, especially in challenging test environments.
4.2 Shifted Random Overlay Augmentation
The random overlay augmentation [3
used in existing approaches augments input images through a linear interpolation with a natural image E HC* randomly sampled from the Places 46 dataset that contains 1.8M diverse scenes:
, (s;e) (as + a)o,..... + a)o,), D.
(4)
5

---

(4 wi)
W
Shifted
Shifted Crop
4
of
Dataset
Sample Natural Image c
Sample Reglons to be Crooped
Image Frames
Hf 1)
(1-a)
W
02
Aupmented State 2500 (x:
State (O), 0g)
Figure 3: Shifted random overlay (SRO) augmentation for data regularization. To inject random dynamics into the backgrounds of RL input images, we generate multiple cropped patches in a shifted manner from sampled natural image and interpolate them augment the input images.
where s is state that consists of a sequence of images (01...., On), O, RC D is dataset and a common choice for the interpolation coefficient augmentation uses the same natural image for all stacked frames. Contrarily, we introduce shifted random overlay (SRO) augmentation to inject the task-irrelevant dynamics into the training images, which is depicted in Fig. Considering a maximum shift length / for each shifting and a stacked frame number n, this method first samples a natural image E E C x from the dataset D, where (Ar, () (H + 2(n 1)4, W + 2(n - 1)1). Then, we crop n shifted patches of E HCCH*M from to augment each frame in shifted way. Given upper left corner coordinate (h1, w1) of H x W sized center crop at the center of e, the coordinates (h;, w;) of the upper left corner of ef are selected follows:
0.5. The original random overlay
(h,, w.) = (h1 + dh(i - 1), wi + dw(i - 1)),
i= 1,.,,...
(5)
where dh, dw unif(-1,1] Finally, using the cropped images of from (h4, wi) in the shifted random overlay augmentation is defined as:
-SRO (s; 1, = (aef + (1 a.)o,,..................,.. as + a)on),
(6)
where we adopt a 0.5. This augmentation provides implicit advantages when training RL agents. Firstly, the inclusion of background elements in motion, independent of the task, enables the RI. agent perceive that rewards solely associated with changes in the genuine task object. Consequently, the agent trained to concentrate on the task object, disregarding the movement of background elements, thereby alleviating the issue of observational overfitting. Secondly, this augmentation method generates training data akin to real environments with dynamic backgrounds, enabling the agent to be robust in such conditions and enhancing generalization capability.
4.3 Simple Framework for Generalization in Visual RL (SimGRL)
Built on the SVEA 12 framework that leverages the data-mixing strategy between weak and strong data augmentations and computes the target Q-value using clean images for stability in SAC, we propose a Simple framework for Generalization in visual RL (SimGRL) under dynamic scene perturbations, which integrates the proposed two strategies. For the strong augmentation, we utilize the shifted random overlay instead of the original random overlay without shifting while the random shift (40) is employed as the weak augmentation. Considering that the clean images 8, and 8 already have weak augmentation, i.., random shift, applied by default, the critic loss for SimGRL is defined as follows:
(0) F.e.eemee (st, g/g2/2 + (1- SRo (5; g2 (7)
where (-.-)) is computed by Eq. (3). gipt= = + 796($1-1, d²), a' TT(-1f((S(41)) and Ø is the parameter of the target network that is updated by the exponential moving average (EMA). The strong augmentation SRD (:: .) the shifted random overlay Eq. (6). Consistent with the SVEA, we adopt the data-mixing coefficient 3 0.5. When training the actor TT. we leverage solely clean
images like the existing methods 122 13 The overall framework of SimGRL is illustrated in Fig.
and the algorithm is summarized in Appendix C

---

Concatenation
Stop-gred
Tarpet Encoder
Target
Trrppt Covaken
Ap-1
Figure 4: Overview of the Simple framework for Generalization in visual RL (SimGRL) under dynamic scene perturbations. Differences from SVEA are marked red.
4.4 Task-Identinn miss-lddetiificatin (TID) Metrics
To quantitatively evaluate the capability to identify task-relevant objects in each stacked frame as salient, we introduce novel metrics, referred to as lask-Identification (TID) score and variance, based on the p-quantile attribution mask Mp(Qe. St, a4) [2] described in Section [
TID Score TID score measures how much the model identifies the task object's pixels across stacked frames, which is defined as:
Nobjse
Nobjs
(Nobjac Nobj x Nas
TIDs V Novi NM
(8)
where Nobj is the number of task object's pixels in input images, Na is the number of pixels in attribution masks Mp. Nobj. the number of task object's pixels included Mp. and p is quantile value for thresholding the attribution map. Note that all numbers consisting of the TID score are counted across the full consecutive frames. Eq. (8. the first term quantifies the model's ability to identify the pixels of the task object, while the second term quantifies how accurately the model identifies the task object's pixels. These two terms trade-off depending on size of the quantile value and are upper-bounded by 1, thus leading an upper-bounded TID score by 1. With model that perfectly identifies all task pixels, this upper-limit value can only be achieved along with an optimal o value, which is the quantile of the number of the task object's pixels in frames. This Noti value can be computed by p 1 [nxCxx xXJ where is the number of frame stack. We explain the impact of the p value on this TID score Appendix
F2
TID Variance TID variance measures how discriminatively the model distinguishes the task object's pixels in each frame, which defined as:
TID Var[100 (TTD), TIDS,TIDS... -IDD...,TTD?))
(N )2 Bay Ny."
where TID's =
Nó obj' No and N obj are individually counted at each frame. Since
each TID5' is upper-bounded by 1, the values of TIDyar become too small for meaningful comparison. Therefore, we multiply TIDs' ' by 100 to obtain a variance with a more appropriate scale for comparison.
5 Experiments
In this section, we present the experimental results of SimGRL on the DMControl-GB [33 video benchmarks ('Video Easy" and 'Video Hard') at 500K simulated training frames demonstrate the generalization capability under dynamic scene perturbations. To tackle vision-based control tasks with continuous action space, we employ the SAC [ a backbone RI. algorithm, and compare SimGRL against current state-of-the-art
Training
Easy
Data Augmentation
Video Hard Level
Figure 5: Experimental setup. We evaluated the zero-shot performances for test environments with dynamic background perturbations.
7

---

Table 1: Generalization performance on video benchmarks from DMControl-GB [3] We report the results on mean and standard deviation over 5 seeds. The scores of the comparison methods were taken from their respective papers, and in cases where the scores were unavailable, they were obtained through our implementation using the official codes. A indicates the difference with second best.
DMCostrol-G GB
SODA
SVEA
SGQN
CC2A
SimCRL
Walker,
608+92
747-21
768-38
819+71
868463
910-34
913-38
918+220
910=21
(0.8%%
Walker, Stand
389=131
879264
926-=30
955-113
961
913#66
955-9
970=223
968:26
973-4
Ball In Cup. Cauch
192+157
363-158
380- 188
875+-56
101
855+556
950-24
911==40
963+-26
964-7
+1 (0 156
Video
Campole, Swingup
898+60
473+-44
459+88
758+622
782+27
671+-57
761+28
762-88
788+224
838-35
-50 (65))
Finger, Spin
2661166
516.1113
599162
695.99
808.35
746118
956.28
717_51
912.669
983 2
-27 (39)
Cheetah, Run
73::18
160::16
268:10
251
289:-35
56
314::49
317:16
(995)
Walker, Walk
122447
80+10
121+-52
312+-2
385+63
292+133
730+21
583-559
687+18
773-31
-34 (5%6
Hard
Walker, Stand
251+557
229+455
252+557
771+-83
834+-46
595+-56
=51=224
744-62
395+335
932-117
(37,555)
Cup.
101.137
1001440
327.1100
40311774
782.157
19
(125)
Carppole, Swingup
158+17
1524-29
136+-79
429-66
393+-45
308+144
544-43
575-37
4724-24
+183 (3496)
Finger, Spin
13110
39.120
18
302141
335.588
256.125
$22.124
277
8191.38
864_12
-12
Cheetah, Run
14
21.19
49
130124
112.112
67.123
157.169
91.146
168.16
301 7
(793)
methods for visual RL generalization including RAD [19] DrQ 40)
SODA 13] SVEA 12 TLDA
(2), EAR 44) and CG2A [23] For weak augmentation, RAD and SODA utilize random crop, while the others employ random shift, except for SAC, which does not use any data augmenta- tions. In addition, all competitors leverage the original random overlay augmentation without shifting for strong augmentation except for RAD and DrQ, which use only weak augmentations. As default for SimGRL, we used the random shift augmentation [40] and denoted the images with only this weak augmentation as clean images with the data distribution of the training data. Additionally, we employed the proposed shifted random overlay (SRO) for strong augmentation in SimGRL and denoted images together with this augmentation as augmented ones. Implementation details are described in Appendix [ and further experimental results on other benchmarks of Distracting Control and robotic manipulation 11 are provided Appendices B.7 and B.8 respectively.
Suite 34
5.1 Results on DMControl-GB
We evaluated the zero-shot generalization performance 6 tasks in 'Video Easy" and 'Video Hard' benchmarks from DMControl-GB [13]. As illustrated in Fig. [5] the easy version shares certain structures with the training environment such as the ground and shadow, while the hard version shares nothing other than the agent's 's object by replacing all backgrounds with distracting videos. In Table [ SimGRL demonstrates state-of-the-a performance in 4 out of 6 tasks in the Video Easy benchmark and achieves comparable performance in the remaining two tasks, including 'Walker, Walk' and "Cheetah, Run'. On the other hand, our approach shows outstanding performance in all 6 tasks at the Video Hard level, where SimGRL achieves performance gain by 15% on average compared to SGQN. Specifically, SimGRL outperforms existing methods by a significant margin 'Cartpole, Swingup' and 'Cheetah, Run', which were previously difficult to solve. We suggest that the reason for this lies in the fact that 'Cartpole, Swingup' requires detailed identification of salient objects, while 'Cheetah, Run' needs to address the observational overfitting. Both issues are mitigated in SimGRL. We provide the training curves in Appendix B.6
Computational Efficiency Our method has the advantage of im- proving generalization performance without any extra models or auxiliary losses. We compare SimGRL with SGQN (2), the existing state-of-the-art method in the Video Hard benchmark. In experi- ments, SimGRL can achieve a throughput of 9.54 FPS, which is 1.55x efficient compared SGQN's 6. FPS on a single NVIDIA
DMControi-GB (Video Hard)
800
Average Return
600
400
200
TITAN RTX GPU. Fig. 6 shows the training curves for the zero- shot test performances over the wall-clock training time, where the averaged performances across the 6 tasks in Table [are presented. In this figure, SimGRL requires 44% less wall-clock training time than SGQN to reach the same 500K frames for training, demon- strating the high computational efficiency of SimGRL. is worth noting that SimGRL has achieved considerable enhancements in both training efficiency and test
SGION SiMGAL (Ours)
0
5
10
15
20
25
Hours (for 500K Frames)
Figure 6: wall-clock time axis.
Training curves at
8

---

performance compared to the SOTA method, regardless of the typical trade-offs in terms of efficiency and performance.
Ablation Study To verify the effectiveness of the proposed reg-
DMControl-GB (Video Hard)
800
ularizations, we compare the ablation variants with the SVEA base- line. Fig. [ shows training curves for zero-shot test performances averaged across the 6 tasks of DMControl-GB at the Video Hard benchmark as function of the number of stepped training frames. This result indicates that the additions of each regularization to SVEA can remarkably improve the generalization ability of the model. In particular, we emphasize that each of the two regulariza- tions leads to sufficient improvement, where all variants of SimGRL achieve, on average, nearly 82% better performance than SVEA. Full experimental results for each task are provided in Appendix B.1 and further in-depth discussions on the effectiveness of each regularization are described in Appendix
Average Return
600
400
Beseiin (SVEA)
200
SImGRL-F (SVEA-Feat Stack)
SimGRL-S (SVEA+SRO)
SiMGAL (SVEA+
Number (x10
Figure 7: Training curves for ablation variants.
B.2 B.2
5.2 Analysis with TID Metrics
To analyze the two potential overfitting problems, namely imbalanced saliency and observational overfitting, we evaluate the TID metrics, including the TID score and variance. For clarity, we compare three methods of DrQ 40). SVEA [1), and SimGRL as the former serve as baselines for the latter. We analyze two representative tasks, namely 'Cartpole, Swingup' and 'Cheetah, Run', where each problem is prominently observed. Full evaluations for all algorithms and tasks are provided in Appendix F3 The left plot in Fig. depicts that the overall distribution of training scores of the TID metrics is divided into three regions. In this plot, both DrQ and SVEA exhibit results divided into regions, either "low score and low variance' 'middle score and high variance', depending on the task, indicating that different phenomena appear in the two tasks. The 'low score and low variance" region be interpreted observational overfitting, which does not identify the correct object in all frames. On the other hand, the 'middle score and high variance' area suggests that the correct object is accurately recognized only in certain frames, implying imbalanced saliency. We note that although SVEA 12] incorporates a data-mixing strategy with strong augmentation to enhance the generalization capability of DrQ 40). both problems are still observed with only minor improvements over DrQ. In contrast, SimGRL reports comparatively high scores and low variances for both tasks, indicating that it can effectively identify task-relevant objects regardless of the type of task. The middle plot shows a positive correlation between training and test TID scores at Video Hard from DMControl-Cis [3. This suggests that high discrimination ability during training can result in high discrimination ability during testing. Finally, the right plot shows the generalization gaps, which are computed by the performance diffcrences between training and testing, against the TID scores. This suggests that high discrimination ability of the task object can lead to improved generalization capability.
'Variance' vs 'Score'
'Test' Vs 'Train' (Score)
'Generalization' 'Score'
0.6
800
TID Varaance (Train)
300
0.5
Generalization Gap
600
250
0.4
200
150
100
200
0.1
0.0
0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
TID Score (Train)
TID Score (Train)
TID Score (Train)
DrO (Cartpole) (Chestah)
SVEA (Cartpole) (Cheetah)
SimGRL (Cartpole) SimGRL (Cheetah)
Figure& 8: (Left to Right) Plots for the training TID variance, test TID scores, and generalization gap against training TID score.

---

Related Works
The field of domain generalization for RL has garnered considerable attention in recent years where various approaches aim enhance the robustness of policies against visual changes. Á promising approach is adapting the policy to a test domain. For example, PAD [TT] suggests adopting a self-supervised task to obtain free training signal during deployment. On the other hand, one method, proposed in [36), involves using randomly simulated RGB images. Similarly, [27] train domain-adaptive policies by randomizing dynamics during the training phase. Several works explore
data augmentation techniques to improve policy generalization capacity [19 40) 22)) For instance, RAD [19] and DrQ [40) achieve significant improvement through random crop and shift, while DrAC [9 automatically identifies the most effective augmentation with regularization terms for the policy and value function. Instead of relying solely on augmented data for policy learning, SODA [3] aims to decouple augmentation from policy learning by using soft-augmented data for policy learning and strong-auge -ooognnegeeeeed data for auxiliary representation learning. In recent development, SVEA [[2] designs a stabilized Q-value estimation framework to address instability issues under strong data augmentation in off-policy RL, while DBC (43) uses bisimulation metrics to learn representation that disregards task-irrelevant information. VAI (39] extracts a universal visual foreground mask to provide an invariant observation to RI. Similarly, several methods leverage salient masks to encourage the agent to focus import pixels. For example, TLDA 42 attempts to only augment the task- iask-irrelevant pixels using masks obtained from the Lipschitz constant of the policy while SGQN 2 proposes saliency-guiee -guieed self-supervised learning using gradient- based saliency mask. EAR 4 attempts to learn environment-agnee -ggnosti representations to enhance the robustness of policies against visual perturbations. On the other hand, several methods such TIA 77.. DRIBO 66. and RePo (47), leverage model- based RL through a recurrent state-space model (RSSM) IO 9 to explicitly learn latent representations that focus task-relevant features while discarding task-irrelevant ones. Our work focuses on improving the generalization ability of the RI. agent based on implicit regularization approaches.
7 Conclusion
In this paper, we identified two key issues that degrade generalization in vision-based RL. particularly under dynamic scene perturbations, by employing a gradient- based attribution mask. To resolve these issues, we proposed a simple framework involving an architectural modification to the encoder and new data augmentation. The proposed SimGRL algorithm has achieved state-of-the-art results on Video Hard environments in DMControl-GB. outperforming existing methods that have yet to address these challenges. Additionally, introduced TID metrics quantitatively evaluate the ability to discriminate task-relevant objects of RL agents. Using these metrics, we demonstrated that improving the identification of task-relevant objects can enhance the generalization capability of the vision-based RL agent.
Acknowledgments and Disclosure of Funding
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the government of Korea (MSIP) (NRF202IRI A222006703, by the Basic Research Lab Program through the NRF of Korea (RS-2023-002223855) and by the Yonsei Signature Research Cluster Program of 2024 (2024-22-0161).
References
[1] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1et..9912008800 2019.
[2] David Bertoin, Adil Zouitine, Mehdi Zouitine, and Emmanuel Rachelson. Look where you look! saliency-guided qenetworks for generalization in visual reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
10

---

[3] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient- based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pages 839-847. IEEE, 2018.
[4] Hyesong Choi, Hunsang Lee, Seongwon Jeong, and Dongbo Min. Environment agnostic representation for visual reinforcement learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[5] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting deep networks by decorrelating representations. arXiv preprint arXiv:1511.0606 2015.
[6] Jiameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi-view information bottleneck. In International Conference on Machine Learning, pages 6074-6102. PMLR, 2022.
[7] Xiang Fu, Ge Yang, Pulkit Agrawal, and Tommi Jaakkola. Learning task informed abstractions. International Conference on Machine Learning, pages 3480-3491. PMLR, 2021.
[8] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870 PMLR, 2018.
[9] Danijar Hafner, Timothy Lillierap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. In International Conference on Learning Representa- fions, 2020.
[10] Danijar Hafner, Timothy Lillicrap. Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages
PMLR, 2019.
[11] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations, 2021.
[12] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in neural information processing systems, 1833680-399,, 2021.
[13] Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 1361 1-13617. IEEF, 2021.
[14] Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal, Mohit Jain, and Xiaolong Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, (2):3046-3055 2022.
[15] Leslie Pack Kaelbling, Michael I. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
[16] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola. Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1705:::1000069500 2017.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning, pages 1639-5650. PMLR, 2020.
[19] Misha Laskin, Kimin I Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances neural information processing systems, 33:19884-19895 2020.

---

[20] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In International Conference on Learning Representations, 2020.
[21] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, (7(1):1334-111 2016.
[22] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 100.::/799.0971 2015.
[23] Siao Liu, Zhaoyu Chen, Yang Liu, Yuzheng Wang. Dingkang Yang. Zhile Zhao, Ziqing Zhou, Xie Yi, Wei Li, Wenqiang Zhang, et al. Improving generalization in visual reinforcement learning via conflict-aware gradient agreement augmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 23436-23446, 2023.
[24] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu. Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. In International Conference Learning Representations, 2017.
[25] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 131.5500, 2013.
[26] Karl Pearson. Vii. mathematical contributions to the theory of evolution. iii. regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, containing papers of mathematical or physical character, (187):253-31,, 1896.
[27] Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. 10. IEEE, 2018.
[28] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint 171.4:171. 2017.
[29] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5402-5415. Curran Associates, Inc., 2021.
[30] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[31] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[32] Xingyou Song. Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In International Conference on Learning Representations, 2020.
[33] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.68066 2014.
[34] Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting con- trol suite-a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:1101.02222, 2021.
[35] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint
2018.
12

---

[36] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In 2017 IEFF/RSJ international conference intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[37] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. com_tro:::: Software and tasks for continuous control. Software Impacts, 6: 600022 2020.
[38] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster in starcraft using multi-agent reinforcement learning. Nature, 575(7782)3350-334 2019.
[39] Xudong Wang, Long Lian, and Stella X Yu. Unsupervised visual attention and invariance for reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6677-6687, 2021.
[40] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. International Conference on Learning Representations, 2021.
[41] Denis Yarats, Amy Zhang. Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 10674- 10671, 2021.
[42] Zhecheng Yuan, Guozheng Ma, Yao Mu, Bo Xia, Bo Yuan, Xueqian Wang, Ping Luo, and Huazhe Xu. Don't touch what matters: Task-aware lipschitz data augmentation for visual reinforcement learning. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 3702-3708. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.
[43] Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine Learning invariant representations for reinforcement learning without reconstruction. In Inter- national Conference on Learning Representations, 2021.
[44] Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv: /004./88000000 008936693, 2018
[45] Zheng Zhao, Lei Wang, and Huan Liu. Efficient spectral feature selection with minimum redundancy. In Proceedings of the AAAI conference on artificial intelligence, volume 24, pages 673-678, 2010.
[46] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence,
2017.
[47] Chuning Zhu, Max Simchowitz, Siri Gadipudi, and Abhishek Gupta. Repo: Resilient model- based reinforcement learning by regularizing posterior predictability. Advances in Neural Information Processing Systems, 36:32445-322 -24677, 2023.
[48] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinay Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357-3364. IEEE, 2017.
[49] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey. et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1435 Chicago, IL, USA, 2008.
13

---

A Implementation Details
In this section, we describe the implementation details used for SimGRL. For fair comparison, we followed the same architectural designs and hyperparameters used in [13], with only a minor difference in the encoder. For the feature-regularizee encoder, we used 3 layers for the image encoder f! with 16 channels and 8 layers with 32 channels for the feature encoder f², leading to the encoder with total of 11 layers that the same layer number as the encoder used in [13]. Subsequently, we projected the convolutional features of the last layer to 100-dimensional linear vectors, which are fed into actor and critic heads with 1024-dimensional hidden layers. We updated the actor parameters by freezing the encoder, and the target network parameters using exponential moving average (EMA) with rate of 0.01 for the critic head and 0.05 for the encoder. Both actor and target networks were updated every 2 critic updates. We used minibatch size of 128, stacked 3 image frames each of size 3 x 84 x 84 as input for RL, and employed the Adam optimizer [17]. For the maximum shift length L in the shifted random overlay augmentation, we employed L 6, which leads to 3 x 108 x 108 size of a natural image before shifted cropping. In Fig. / we present examples of augmented images by the shifted random overlay. This figure exhibits that the proposed augmentation approach injects randomly moving backgrounds into the training images. All hyperparameters are summarized in Table. 2
Table 2: Hyperparameters used in DMControl Suite.
SAC hyperpurameters
Value
Replay buffer cupacity
500,000
Number of training steps
500,000
Frame size
3x84x84
Stucked frames
Evaluation episodes
30
Random shift
Up to +:4 pixels
Action repeat
2 (Finger. Spin)
8 (Cartpole, Swingup)
4 (otherwise)
Minibatch size Discount factor (t) Optimizer
128
0.99
Adam (3) = 0.9, 08 0.999 for 6 and d)
Adam 0.5, 82 0.999 for a SAC)
Learning rate
Ie (6 and d))
le- 4 (a of SAC)
Actor (0) update frequency
Target (f) update frequency
2
Target (1) EMA rate
0.01 (critic)
0.05 (encuder)
Number of conv layers
f Imuge encoder)
8 (f Feature encoder)
Number of conv filters
16 (f), Image encoder)
32 (f², Feature encoder)
Projection dim
Hidden dim (MLP heads)
1024
Initial temperature
0.1
SVEA coefficient (2)
Weak augmentation
Random shift
Strong augmentation
Shifted random overlay (SRO)
Maximum shift length (1) in SRO
Up to + pixels
(G.4)
Figure 9: Examples of augmented images using shifted random overlay (SRO).
14

---

B Additional Experimental Results
B.1 Detailed Results on Ablation Study
We provide detailed results of the ablation study in Table 3 For DrQ, we utilized [K = 1, M = 40 In this table, by introducing weak augmentation using random shift into the vanilla SAC [8].
DrQ demonstrates highly improved performance in image-based RL at the training level. While
DrQ reports the best results in most tasks at the training level, the performance drastically decreases in testing levels, implying overfitting to the training environments. SVEA 12] introduces a data-
mixing strategy to enhance training stability under strong data augmentation, thereby improving DrQ's generalization capability in distracting backgrounds using random overlay [13]. However, the performance of SVEA still falls behind in challenging environments such as Video Hard. To address this issue, we proposed the feature-level frame stack and shifted random overlay augmentation, resulting in significant improvements compared to the SVEA baseline in Table especially in the Video Hard environments. It is noteworthy that each of the proposed regularizations demonstrates significant improvements compared to SVEA, where SimGRL-F and SimGRL-S apply the feature- level frame stack and the shifted random overlay augmentation to SVEA, respectively. Especially, all of SimGRL's variants reach the saturated performance in several tasks at the Video Hard level except for 'Walker, Walk' and 'Cheetah, Run'. For more complicated tasks such as 'Walker, Walk' and 'Cheetah, Run' tasks where performance saturation has not yet been reached, the integration of the regularizations (i.e., SimGRL) clearly shows the performance improvement. Fig. 10 shows the training curves of ablations for each task at Video Hard.
Table Results of the ablation study in DMCotrol-GB video benchmarks. In strong data augmenta- tion, RO and SRO indicate the original and shifted random overlay data augmentations, respectively. Each score represents the mean and standard deviation over seeds.
Algorithm
Image
Prame Stack
l'eature
Random Shift
Aug
RO
Strong Aug.
SRO
Walk
Walker, Stand
Ball In Cup.
Catch
Campole, Swingup
Finger, Spin
Cheetah,
Run
257.1.49
$61.123
807.110
640-1117
214.126
DrQ
943-+3
973+2
975--5
878-2
982+4
463-111
Train
SVEA
923::14
97244
974:5
872-4
976:15
196::10
SimGRL-F SimGRL-S
903-425
972:43
972:+4
865=7
982++
310+24
901::36
973:133
974:13
868113
98242
314218
SimGRL.
923:117
971.12
969.1.
8732-5
986.12
322.123
SAC DrQ
245=165
389-131
1922:157
398160
206:1169
73:188
Video Easy
747.1:21
926.130
380 188
459181
599.162
270:116
SVEA
819++71
961+8
871+1006
782+227
808 808+33
251++17
SimGRL-F SimGRL-S SimGRL
947444
878:432
973-55
962=10
826+25
962+-11
301+223
910:121
973:14
96427
8381:55
98312
317:116
SAC DrO SVEA
122:147
231:577
101=37
158+17
13:210
75:214
Video Hard
121:152
252-:57
100:440
1361:29
38:113
49:113
385:163
8341446
403
393.145
335.1588
112:112
SimGRL-F
7191334
878121
264113
SimGRL.S
727
91849
861-:3
708--4
886-14
269-111
SimGRL
773:t31
932:177
902=19
727=23
864+:12
301+:
Walker, Walk
Walker, Stand
Ball Cup. Catch
1000
800
800
800
600
600
200
Cartpole, Swingup
2000
Finger: Spin
430
Cheetah, Run
500
300
600
230
400
40c
200
30C
0
0
Number tramees Gc20")
Number Framms Ixte")
Numberr of Fremiz (x205)
Baseline (SVEA)
SmCRL-F
SimGRL-S ISVEA+ SRO)
SimGRL (SVEA+Both)
[SvEA+Feat Stack)
Figure 10: Trainig curves of ablation variants including the SVEA baseline at the Video Hard level.
15

---

B.2 Discussion on Impacts of Proposed Regularizations
We discuss the impacts of the proposed regularizations based on the attribution masks and the TID metrics. First, we investigate the impacts in terms of the imbalanced saliency in the 'Cartpole, Swingup' task. In the left plot Fig. IT both SimGRL-F and SimGRL-S show increased TID scores compared to SVEA. However, they exhibit different effects on TID variance as SimGRL.- significantly reduces it but SimGRI--S rather increases it. This indicates that the imbalanced saliency remains in SimGRL-S while SimGRL,-F alleviates this issue. Therefore, the performance improvement of SimGRL-F can be interpreted as a mitigation of the imbalanced saliency problem, while the performance improvement of SimGRL-S is a bit ambiguous. For the reason of the result with SimGRL-S, we suggest that despite the persistence of imbalanced saliency, the explicit injection of dynamic background elements might have contributed to improved generalization in such conditions. Furthermore, even though it does not distinctly discriminate task objects between the stacked frames, the increased TID score of SimGRL-S can be attributed to its improved accuracy in identifying recognized objects in certain frames, as depicted in Fig. 12 This implies that the shifted overlay augmentation implicitly encourages the model to identify task objects more accurately. As result, SimGRI., which integrates both architectural and data regularization, demonstrates that its agent can identify task-relevant objects across stacked frames with greater accuracy and discrimination.
In the right plot of Fig. I all of the SimGRL variants demonstrate remarkably increased TID scores in the 'Cheetah, Run task. Since the 'Cheetah, Run" task has less movement of task objects than the Cartppole Swingup' task, differences in TID variance values for this task are not effective. Therefore, the core issue in this task is the observational overfitting [32 to ground shadow rather than imbalanced saliency between consecutive frames. The increase in the TID score implies an enhanced ability to identify task-relevant objects, which is demonstrated in Fig. 12 this shows that the application of the shifted random overlay augmentation encourages the agent to focus the task object while disregarding background elements. On the other hand, it is noteworthy that the frame stack at the feature level also contributes to solving this problem. We suppose this is because encoding from single image without temporal information allows the model to concentrate on each image's most salient visual features. Consequently, by integrating those strategies, the proposed approach, SimGRL, effectively addresses the two problems highlighted in this paper that hinder generalization in visual RL.
As expected,
Cartpole, Swingup
Cheetah, Run
350
350
300
300
TID Veriance (Irain)
250
250
200
Variance
200
150
150
100
100
50
50
0
0.4
0.2
TID Score (Train)
TID Score (Train)
Baseline (SVEA)
SimGR--F
SimGRL-S (SVEA+SRO)
SimGRL
(SVEA+Feat Stack)
+VEA+Both)
Figure 11: Plots of 'TID variance (Train) vs TID score (Train)' for ablation variants.
16

---

Observation
SVEA
SimGRL-F
SimGRL-S
SimGRL
t 2
Observation
SVEA
SimGRL-F
SImGRL-S
SimGRL
t 2
Figure 12: Examples of attribution masking for ablation variants.
B.3 Impact of Number of Layers in Image Encoder
Walliar,
Walker,
Wallker; Walk
802
500
30-
600
600
200
Number of Layers Image Encocer
Number of Layerr Imagge Encodee
Number of Layers Image tricnder
Figure 13: Impacts of the number of layers in image encoders.
While maintaining the number of layers the entire encoder 11, we conducted the ablation study for the number of layers in the image encoder. To consider only the impact of the layer number, leveraged the SimGRL-F model which employs the original random overlay without shifting for the strong augmentation rather than the shifted version. Depending the number of layers in the image encoder, we investigated test rewards, generalization gaps, and throughputs in the 'Walker, Walk' task, where the Video Hard domain employed for testing environments. The generalization gap was evaluated by the difference between the training and testing performances, and the throughput a single NVIDIA TITAN RTX GPU. The
was computed the FPS processed in the simulator results presented in Fig. 13 demonstrate that the incorporation of the feature-level frame stack, facilitated by the image encoder, enhances the model's generalization capabilities in dynamic scenes. is noteworthy that even if the number of image encoder layers is only I, the generalization gap is remarkably reduced. Additionally, the generalization gap decreases when the number of layers in the image encoder is increased, indicating that incorporating feature maps with higher abstraction enables the improvement of generalization. However, despite the improvement generalization performance, there does not seem to be a significant increase in practical test performance, implying reduction in the training performance through the mitigation overfitting. This can also be attributed to the diminished depth of the feature encoder, which is crucial for encoding temporal information. Moreover, increasing the layer number the image encoder results decreased throughput, primarily due to the increased computational burden associated with processing individual frames. Considering both generalization capability and computational efficiency, we selected 3 layers for the image encoder and 8 layers for the feature encoder for the experiments.
17

---

B.4 Different Data Augmentations
(h. MG)
Irrewwlrry Shifted Patches
Shifted Crop
11
4
Dataset
Sample Natural
Imegularly Sample Regions be Cropped
Image
H 2(x 13
R +
0g
Augmented State
State (O,, 0g)
p--30 (xt)e)
(a) Irregularly Shifted Random Overlay (I-SRO)
Sampled Arbitrory Natural Images
x
62
Fg
Dataset
+
Image
(1-4) a) x
Augmented State A-5R0 (g)5))
State s (gg,0gx 0g)
(b) Arbitrarily Stacked Random Overlay (A-SRO)
Figure 14: Additional data augmentation approaches for ablation study.
To investigate the impact of differently augmenting the stacked frames, we provide additional experiments under two types of additional augmentations: (1) randomly cropped patches from the same image and (2) completely different images. We denote the first method irregularly shifted random overlay (I-SRO) and the second method as arbitrarily stacked random overlay (A-SRO). While the SRO augmentation crops patches in regularly shifted manner as Eq. (5). I-SRO irregularly crops patches from sampled natural image e. In contrast to Eq. (5). the coordinates (h;, w;) of the upper left corner of each cropped patch become as follows:
(h4, w.) (h1 dh4, w1 + dw.),
= 2,.,,,
(10)
where dh, dw; unif{-1,4 On the other hand, A-SRO samples different natural images E; D with x sizes and augments each RL frame using the different images. The processes of I-SRO and A-SRO are illustrated in Fig. [4
For clarity of the effect for each augmentation, we do include the feature-leve frame stack and compare them in the video hard level with SRO (i.e., SimGRL-S in the paper) that leverages regularly shifted cropped patches from the same image for data augmentation. Table 4 shows that augmentations using irregularly shifted patches (I-SRO) or completely different patches (A-SRO) across the consecutive frames can achieve comparable performance with SRO using the regularly shifted patches, implying that augmenting each frame in a different manner (i.e., employing SRO, I-SRO, and A-SRO) can achieve significant performance improvement over augmenting each frame uniformly (i.e., RO) for visual RL generalization under dynamic scenes. Among such augmentations, we have employed the (regularly) shifted random overlay (i.e., SRO) as our primary approach for data augmentation considering its superior averaged performance.
18

---

Table Performance on DMControl- at video hard level for different strong data augmentations.
The scores were evaluated over 5 seeds. Percentages indicate variations compared to RO.
Environment
RO (=SVEA)
SRO (ESimGRI-SS)
I-SRO
A-SRO
Walker, Walk
385++63
727++36 (489%)
715+331 (+86%)
732 -2 (90%)
Walker Stand
8341:46
918 (+10%)
91848 (10%)
910+12 (+9%)
Ball In Cup. Catch
403=1744
861#36 (+1 (+1 14%)
8354-42 (107%)
8324:28 (+ (+10%%
Caripole, S Swingup
393:145
70844 (+80%)
69719 (477%)
70545 (+79%)
Finger. Spin
335+58
88644 (+164%)
829:220 (147%)
804416 (+140%)
Cheetah, Run
112412
269111 (+140%)
2324:17 (4107%)
237:21 (+111%)
B.5 Adoption of SGQN
Based on its architectural simplicity and absence of additional auxiliary losses beyond the actor and critic losses, we adopted the SVEA [2] algorithm as our baseline clarify our contribution. However, our approach that utilizes two types of regularizations can be seamlessly integrated with any other algorithms a plug-and-play manner. On the zero-shot evaluation
video hard test environments, we present the experimental results of SGQN 2 F (Feature-level frame stack), SGQN S (Shifted random overlay), and SGQN S + F in Table This table shows that the proposed regularizations also contribute to improved generalization for other methods, such as SGQN, demonstrating the generality of our method.
Table 5: Performance on DMControl-GG at video hard level for SGQN applying the proposed approaches. The scores were evaluated over 5 seeds. Percentages indicate variations compared to SGQN.
Environment
SGQN
SGQN +
SGQN S
SGQN + F+
Walker, Walk
739-21
773+18 (+4.6%)
786-22 (+6.4%)
811+13 (+9.7%)
Walker ,ttand
851-244
8851.34 (+4%)
906_15 (+6.5%)
913 (17.33%
Ball In Cup. Catch
782-57
826:224 (45.6%)
894438 (14.3%)
901::12 (+15.2%)
Cartpole, Swingup
$44143
679.134 (+24.8%)
622.31
678.125 (+24.6%)
Finger, Spin
822-24
886+222 (47.8%)
976.77
881=166 (+7.2%)
Cheetah, Run
157_69
2141552 (+36.3%)
2987.19887
311:66 (+100%)
B.6 Training Curves DMControl-GB
In Figures 15
16 and I7 we illustrate the training curves by re-implementing the comparison
algorithms ourselves using available official codes over 5 seeds.
Walker, Walk
Walker, Stand
Cup, Catch
1000
1000
1000
800
800
600
600
400
200
200
200
0
2
Cheetah, Run
Cartpole, Swingup
1000
Finger, Spin
400
Average fretum
800
600
600
300
400
200
200
100
0
3
Number Frames (x10P)
Number Frames (x10
Number Frames (x10²)
SVEA
Figure 15: Training curves on the training level.
19

---

Walker, Walk
Walker, Stand
Ball In Cup, Catch
1000
1000
800
800
600
400
400
Cartpole, Swingup
1000
Finger, Spin
400
Cheetah, Run
800
000
400
100
200
200
NUmber Frames (X207)
Number of Frames (X.10))
NUmber Frames (x107)
SVEA
TLDA
SimGRL
Figure 16: Training curves on the Video Easy testing level.
Walker, Walk
Walker, Stand
Ball In Cup, Catch
1000
800
800
600
400
400
1000
Cartpole, Swingup
1000
Finger, Spin
400
Cheetah, Run
800
800
600
600
200
400
100
200
Number Frames (x20
Number of Frames (x10P)
Number of Frames (x10
RAD
SODA
SVEA
TLDA
SGOON
SimGRL
Figure 17: Training curves on the Video Hard testing level.
20

---

B.7 Results on DistractingCS
(a) Intensity = 0.1
(b) Intensity = 0.3
(c) Intensity 0.5
Figure 18: Examples of DistractingCS benchmark.
We provide the additional benchmark results on the Distracting Control Suite (DistractingCS) 34
where camera pose, background, lighting, and colors continually vary throughout episode. In Fig. 18, we depict some examples of the DistractingCS in intensity levels = (0.1, 0.3, 0.5]. this experiment, we utilized the same hyperparameters used in DMControl-Gii Fig. 19 demonstrates SimGRL's significantly improved generalization performances in low intensities, compared to DrQ and SVEA. In particular, SimGRL is the only method that reported practical performances in the "Ball in Cup. Catch' and 'Finger, Spin' tasks. Furthermore, SimGRL achieves additional performance gains in the 'Walker, Walk, Catch' and "Walker, Stand' tasks. On the other hand, despite the still low performance in 'Cartpole, Swingup', there are clear signs of improvement. These results indicate the robustness our proposed approach not only dynamic background changes but also various forms of distortions, including camera pose variations at moderate intensities. Finally, we note that generalization in extremely distorted environments, such as those with high intensities in the DistractingCS, remains an open problem and will be subject of promising future research.
1000
1000
800
800
Average Return
600
600
400
400
200
200
0.5
Intensity
Intensity
1000
Ball Cup, Catch
600
Cartpole, Swingup
1000
Finger, Spin
500
800
Averogee Peruum
Average Return
Average Return
400
800
300
200
100
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Intensity
Intensity
Intensity
DrO
SVEA
SimGRL
Figure 19: Performance on DistractingCS. Each score reports the mean over seeds.
21

---

B.8 Results on Robotic Manipulation
To verify the generalization capabilities of the proposed approach across various domains, we conducted additional experiments for robotic manipulation tasks [14], where the agents are trained for 250K steps. In these experiments, since the environments introduce color and texture changes
rather than structured distractions, SGQN (2) employed random convolution 120 19 for strong augmentation. However, for SimGRL, we still utilized the shifted random overlay. In addition, for shifting, we employed two frames an input rather than single image setup as in SGQN. Fig. (00 exhibits examples of two robotic manipulation tasks, including 'Reach' and 'Peg in Box', and each of the testing environments. In these examples, SimGRL demonstrates its ability to appropriately capture salient regions, characterized by the robot arm and target positions. Teeporor the experimental results of the Two tasks. Although this benchmark primarily features color and texture changes rather than dynamic background perturbations, SimGRL demonstrates superior results, achieving the best performance in out of 3 tests in the 'Reach' task and in 2 out of tests in the 'eg in Box' task, and the second best in the remaining tests. It is noteworthy that SimGRL, which leverages only a variant of the random overlay, provides comparable performance to CG2A 23 that employs both random convolution and overlay augmentations.
Table 6: Performance on Robotic Manipulation. The scores were evaluated over 5 seeds.
Task
Environment
SAC
SODA
SVEA
SGQN
CG2A
SimGRI
Train
9.7.12
31811
32.210
31.811
39.6
3
-6.1
Reach
Test
120.9116
-30.9-33
-17,6110
14.41:4
28.7=1
32.61:1
+3,9
Test
12
-20.2-29
-2.14:99
31.0133
36.7
31.94.2
Test
.43.2.16
-68.4_30 30
1.4.199
29.2.17
35.4
33.3.1
-2.1
Test Average
-886+8
-3,9931
+6.122
24.946
33.6
32.64
-1.0
Train
-46.777
180.0EE1
177.5#1
183.91
189.9-11
213.6:22
+23.7
Peg in Box
Test
-59 -59.61126
16.9144
-21.1110
-72.0114
155.4-17
181.7.1.5
+26.3
Test
-60.15+10
0.7430
96.8140
110.7-3
157.8-22
138.54:18
-1933
Test
-88...17
73.6.131
40.5
154.617
174.0
191.4.127
Average
-56.247
30.4E31
38,6423
64.4
162,4220
170.5423
Observation
O M,
Observation
O M,
Train
Test
Test
Test
(a) Reach
(b) Peg Box
Figure 20: Examples of Robotic Manipulation Tasks.
22

---

C Algorithm for SimGRL
Algorithm A Simple Framework for Generalization in visual RL (SimGRL).
I: Hyperparameters: Total number of training steps T, mini-batch size N, learning rate 7/ target network update rate X, actor and target network update frequency M, transformation function set V for random shift, natural image dataset D for SRO, critic loss coefficient B, maximum shift length in SRO, replay buffer B, discount factor y.
2: Initialize: encoder and critic parameters 0, actor parameters Φ, target network parameters 0 <-
3: for timestept 1...7' do
4: ag To (-))))
D Sample action from actor (= policy) Sample transition from environment Add transition to replay buffer Sample batch from replay buffer
5: s P(-
6: B (st, as, r(s at).
7:
(sh r(s,, 8 ~
8:
for i I...N
9:
8; (s);
(8(;21), us UM
Apply random shift augmentation
q. r(s;, a,) + +20olf((5) af), of (T((fo((()))
Compute target Q-value
s. suug 5RO (s E
Apply SRO augmentation
12:
end for
(0) N EN (B(Qo(fo(s)) a:) g: 22 + - ()o(os 0g q:
Agt
14:
b Compute SimGRL loss > Optimize encoder and critic for SimGRL loss
15:
0 + 0 - (LQ
16:
if everryy M step then
17:
x ( i 0 10 + (1 X)0
> Optimize actor actor loss Update target network using EMA
18:
end if
20:
end for
Note: fs;); represents f3( (0:-2)
2), fo (o:-1); (o. (o.))); where s, (o, 0z-1, 0z).
D Analysis of Attentions
[Cartpole, Swingup]
[Cheetah, Run]
SVEA
SimGRL
SVEA
SimGRL
Image
Attention Mask
V.
V
04
02
1-2
L-1
1-2
Figure 21: Attention maps were obtained from the softmax of the saliency maps before thresholding by quantile to compute the mask. The attribution masks indicate that SimGRL can effectively distinguish task-relevant objects and backgrounds, implying mitigation of the imbalanced saliency and observation overfitting. Furthermore, the attention maps indicate that SimGRL can focus on the significant parts of the task objects. contrast, the SVEA baseline focuses on the task-irrelevant background parts.
In this paper, we demonstrated that it is important for the RL model not be disturbed by task- irrelevant backgrounds for better policy generalization. To do this, we assumed that the RL agent should at least focus more on the task object than on the background, and employed the attribution masks to analyze whether the trained RI models correctly distinguish the task objects and back- grounds. Additionally, to quantitatively analyze how well RI. models can distinguish between the task object and the background, we proposed the TID metrics.
23

---

about few key points within such objects that are most predictive of the reward. To investigate whether the ability to distinguish between task objects and backgrounds also helps in identifying the key points that influence decisions, we provide attention maps obtained from the softmax of the attribution maps M (Q+, St, az) aQu(s..a. before thresholding for the binarized masks. Fig. 21 3e, shows that the identification ability for the task objects of SimGRI. also leads to focus on the most In practice RL policies may not make decisions based on the entire object. Instead, they could care
important key point parts such the pole and body of the "Cartpole" legs of the 'Cheetah' in the
attention maps. On the other hand, the SVEA baseline incorrectly considers the background features key points.
E
Comparison to Robust RL Algorithms
[Cartpole, Swingup]
DBC
TIA
DRIBO
RePo
SimGAL
Image
Mask
u
Masking
2-
t-1 t-2
t-1
Figure 22: The masking results indicate that DBC, which uses the image-level frame stack, struggles with the imbalanced saliency problem whereas the TIA, DRIBO, and RePo, which feed single images recurrent state-space model (RSSM) [10 9 encoders sequentially, distinctly identify salient pixels of each frame.
In response to the recommendation of the reviewers to investigate whether the pitfalls also apply robust RL algorithms such as DBC 43), TIA 77. DRIBO (o. and RePo [47]. we provide comparative experimental results for these methods. While our work focuses on addressing the generalization issue in a purely model-free vision based RL setting, which requires the frame stack to encode temporal information, the robust RI. approaches aim to explicitly learn robust representations against background distractions by utilizing model-based RI. To encode temporal information, these approaches, except for DBC, train transition dynamics by leveraging complex recurrent neural network (RNN) encoders which take only single frame as input to the encoder, thus applying concept similar to feature-level frame stacking. On the other hand, DBC uses a CNN encoder that takes stacked frames and trains dynamics model that predicts the latent states of the next stacked frames. Fig. 22 shows that methods like TIA, DRIBO, and RePo, which use RNN-based encoders, do not struggle with the imbalanced saliency problem, successfully identifying each salient object across consecutive frames like SimGRL. In contrast, despite employing robust representation learning, DBC still struggles with the imbalanced saliency problem, showing bias toward identifying pixels from the object in the most recent frame salient across consecutive frames.
Cartpole, Swingup (Test)
800
TIA
Average Return
DRIBO
600
SIMGRL
Number of Frames (x10=)
Figure 23: Training curves of DBC, TIA, DRIBO, RePo, SVEA, and SimGRL. In the test stage, we used the video-hard version for the test environments. Compared to robust RL approaches that employ model-basee representation learning strategies, the proposed SimGRL achieves both sample efficiency and test performance even with the simple structure.

---

shows the curves of zero-shot test performances for the 'Cartpole, Swingup' task in the video-hard environment according the number of training frames. For a fair comparison, we applied the random overlay augmentation to the robust RI, algorithms where following to [13] the data augmentation was applied only during the representation learning stage. As anticipated, robust RL algorithms like DRIBO and RePo, which can distinctly identify task-relevant objects, achieved better test performance (except for TIA) than methods that struggle with imbalanced saliency, such as SVEA and DBC. We suggest that the reason for the inferior performance of TIA is that the TIA agent struggles to train the Distractor Model, which explicitly captures the background, due to the more diverse background augmentations compared to the relatively limited video backgrounds from the Kinetics dataset [16] used its original experimental setup. Notably, despite its straightforward structure and absence of additional representation learning stages, the proposed SimGRL achieved both sample efficiency and strong test performance, outperforming robust RI. approaches like DRIBO and RePo that rely model-based representation learning strategies.
Fig. 23
Details on TID Metrics
F.1 Dataset for TID Evaluation
To fairly compare TID metrics of comparison methods the same images, constructed dataset for the evaluation in advance. As illustrated in Fig. [44] we excluded samples that were excessively static because the side effect of imbalanced saliency is diminished in such samples. Additionally, we excluded samples where the task object was not fully visible. Instead, we included dynamic samples to fully evaluate the impact of imbalanced saliency. To this end, the samples obtained from random actions were less dynamic, we selected the images by running SVEA's policies, our baseline algorithm, up 100K steps obtain variety of dynamic samples. Using those images, as depicted Fig. 25 we constructed 60 pairs of images and ground truth (GT) masks of task objects for each training and testing environment in Video Hard. These GT masks
employed to count Nonj and N Novi in Eq. (8) Here, Not represents the pixel number in the GT mask while Nobi is determined by the pixel number the overlapping regions between the GT and attribution masks.
Stacked Frames
Stocked Fromes
Stacked Fromess
Stacked Frames
t-1
Examples of Excluded Samples
(b) Examples of Included Samples
Figure 24: To evaluate the TID metrics on suitable samples, we excluded overly static partially visible images for task objects.
Image
K<
Mazk
π<
T(
Image
Figure 25: Examples in a dataset for the TID evaluation. (First and Second Row) Images from training environments. (Third and Fourth Row) Images from test environments in Video Hard.
25

---

F.2 Impact of Quantile Value
In Fig. 26 the impact of quantile values for the p-quantile attribution mask 2 is illustrated. Additionally, the effect of the p value on the TID score can be explained as follows. When the value of p is too small, the region of mask is enlarged, thus making it easier for task objects to be included the attribution mask. This corresponds to the increased Nobise leading to an increased NobiM in Noks
the first component of the TID score in Eq. ( Simultaneously, this also increases NM. balancing the TID score by decreasing Nobia that is the second component of the 'TD score. Conversely, Nas
large value of p reduces the size of the mask NMs as in Fig. 26, consequently reducing Nobjar On
the other hand, the decreased NM might increase Nabin,
Nur balancing the TID score. Therefore, for the quantile value used to evaluate the TID score and variance, we used the optimal quantile value Nety computed by / p= [3xCxXxW] rather than hand-designed value. However, we used 0.95 in our qualitative illustrations of the attribution masks, empirically provided the most plausible visual representation of a masked region.
Observation
p= 0.9
p 0.95
p optimal
2
1
t --2 2
t-1
Figure 26: Impacts of quantile values used for thresholding.
F,3 Full Results of TID Evaluation
We present comprehensive results of the TID evaluation, including comparison of training TID variance, test TID score, and generalization performance against the training TID score. For the 'Walker' agent, evaluated the TID scores the 'Walk' task rather than 'Stand'. Fig. 27(a) reports higher TID scores and lower TID variances of our approach, indicating that SimGRI. can distinctly identify the salient pixels across the stacked frames for all tasks. In particular, for the BBall In Cup' agent, measuring the TID scores accurately is difficult due to the relatively small size of the salient object. Nevertheless. SimGRL shows remarkably higher TID with lower variance scores than the competitors. These results demonstrate that SimGRL can effectively alleviate the two highlighted issues, 'imbalanced saliency' and 'observational overfitting' observe that the high identification ability of the task object in the training environment can lead to a similar capability in test environments. Finally, Fig. [7(c) shows a positive correlation between generalization performance and the TID score, where smaller generalization gaps correspond to higher generalization performance. For the distributions in Figures [7(b) and [7(c) we present
Fig. 27() we can
Pearson correlation coefficients (26) in Table 7
These results imply that the high identification
26

---

capability for task-relevant objects across stacked frames in input, as represented by the high TID score, can contribute to improving generalization performance.
Viblter: Woll
Cup,
Carppole Swirgip
Einger
500
10:0om (aal
100aco (rak)
10 (980)
Plots of "TID variance (Train) TTD score (Train)'
Eallin Cua, Catch
Cartpole: Setaguo
Roger. Sptn
Cheetai, Run
06
0.8
06
03-
TO 1200 1300
0.-
03
0.4
03
47
01
03
3.5
6-5
0.1
08
6.3
022
5.0
0.
6.2
06
D0
64.
5.0
(b) Plots of 'TID score (Test) vs TTD score (Train)'
Viblier Wbil
Cup,
Carppole, Swirgup
Finger:
700
Comnehuter1. Cat
402
100
282
200
18 sove (Ya))
Seaw (ean))
s000 (Tak)
10 15a
(c) Plots 'Generalization gap vs TID score (Train)"
SimGRL (Ours)
Figure 27: Full results on the TID evaluation.
Table 7: Pearson correlation coefficients.
Correlation Coefficient
Walker, Walk Ball In Cup. Catch
Carppoe, Swingup
Finger, Spin
Cheetah, Run
Avg.
p(T 17g TTD
0.92
0.66
0.62
0.93
0.83
0.79
ATTD GenGap)
-0.56
-0.59
-0.56
-0.64
-0.84
-0.64
27

---

F.4 Further Examples of Attribution Masking
For the existing otate-of-the-art visual RL approaches for generalization, we provide rich examples of the attribution masks and the corresponding masked observations, where we used the images employed in the TID evaluations. Additionally, we used the quantile p 0.95 across all examples. Figures] 28 and 29 depict the examples for training and Video Hard testing environments, respectively.
Observation
RAD
SODA
SVEA
TLDA
SGQN
1
.1
1.6
2.4
28

---

Figure 28: Examples of the attribution masks and masked results for given observations in the training environments.
29

---

SGQN
SimGRL
X
2
+
B.
ALL
30

---

Figure 29: Examples of the attribution masks and masked results for given observations in the test environments of Video Hard.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction include the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification:
The paper has limitations, but those are not discussed in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
32

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper includes experimental result reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification:
The paper includes experiments requiring code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Suubmisioooooioooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting is presented in the paper in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper includes the results that are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper includes experiments compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:://eeurip::: http://uwrr..cccooooccc/caaccaaa
Answer: [Yes]
Justification:
The research was conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no safeguards risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Creators original owners of assets used in the paper, properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "xTc9P85DbO", "reviewer": "Reviewer_z8gS", "summary": "The paper introduces SimGRL, a framework aimed at enhancing generalization in vision-based deep reinforcement learning (RL) under dynamic scene perturbations. It addresses two critical issues in existing visual RL methods: imbalanced saliency and observational overfitting. SimGRL employs architectural modifications to the image encoder and a novel shifted random overlay augmentation technique. Extensive experiments demonstrate SimGRL's superior generalization capabilities, achieving state-of-the-art performance on the DeepMind Control Suite and other benchmarks.", "questions": "How does the diversity of natural images used in the shifted random overlay augmentation impact the generalization performance of the RL agents? Are there specific types of images that are more effective?\nAre there any plans to test SimGRL in real-world environments, and what challenges are anticipated in such scenarios? How does the method handle real-world dynamic perturbations that are not present in simulated benchmarks?\nWhat are the effects of using different numbers of frames in the feature-level frame stack on the performance and generalization capability of SimGRL?\nHow sensitive is the performance of SimGRL to the choice of hyperparameters such as the number of layers in the image encoder and the maximum shift length in the shifted random overlay augmentation?", "limitations": "The architectural modifications and data augmentations, while effective, still introduce some computational overhead, which may impact training times and computational resources required.\nThe method is specifically designed for vision-based RL and may not directly apply to other types of RL tasks without visual components.\nThe shifted random overlay augmentation relies on the quality and diversity of natural images used for augmentation, which may limit its performance in scenarios where such images are not available or appropriate.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Innovative Architectural Modification: The proposed modification to stack frames at the feature level instead of the image level effectively addresses the imbalanced saliency issue, ensuring the agent focuses on spatially salient features in each frame .\nNovel Data Augmentation Technique: The shifted random overlay augmentation introduces dynamic background elements during training, which helps the agent to focus on task-relevant features and ignore irrelevant background changes .\nComprehensive Evaluation: The method is thoroughly tested on multiple benchmarks, including the DeepMind Control Suite and DMControl-GB, demonstrating superior generalization performance compared to state-of-the-art methods.\nQuantitative Metrics: The introduction of TID metrics provides a quantitative way to evaluate and understand the method's effectiveness in identifying task-relevant features, correlating high TID scores with improved generalization performance.\nComputational Efficiency: The method achieves significant improvements without requiring additional auxiliary losses or networks, maintaining computational efficiency.", "weaknesses": "Dependency on Augmentation Quality: The effectiveness of the shifted random overlay augmentation relies on the quality and diversity of the natural images used for augmentation, which may limit its performance in certain real-world scenarios where such images are not available or appropriate.\nAd Hoc approach: the authors offer specific fixes to observed shortcomings in vision-based deep RL in simulated environments, and it remains to be seen if these fixes are an hand-crafted overfitting to specific tasks that won't generalize to other tasks.\nFinally, while the method shows impressive results in simulated environments, its applicability and effectiveness in more realistic real-world scenarios have not been thoroughly validated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your thoughtful comments. We address individual comments in the following.\n\n**[Weakness]**\n\n***\n\n>**[W1] Dependency on augmentation quality**\n\n**(A)**  The effectiveness of the shifted random overlay augmentation may depend on the quality and diversity of the natural images, and the performance may be limited in specific real-world scenarios where such images are unavailable or unsuitable. Nevertheless, we demonstrated in **Q1** that even with fewer and less diverse augmentation data, robust performance can be maintained.\n\n***\n\n>**[W2] Ad Hoc approach**\n\n**(A)**  SimGRL consistently demonstrates robust generalization performance across various distraction environments and task scenarios, including DMControl-GB (Section 5.1 and Appendix B.6), DistractingCS (Appendix B.7), and robotic manipulation (Appendix B.8). Therefore, we do not consider this to be overfitting to specific tasks.\n\n***\n\n>**[W3] Applicability to real-world scenarios**\n\n**(A)** \nBecause it is very difficult to build a safe real-world RL experimental environment using actual robots, existing studies have primarily verified algorithms in simulation environments. Similarly, we conducted experiments in simulation environments to compare our method with existing methods. Nonetheless, applying the algorithm to more realistic real-world situations is crucial for practical use, and we also consider this an important area for future research.\n\n***\n\n**[Question]**\n\n>**[Q1] Impact on the diversity of natural images**\n\n**(A)** We provide the average results of SimGRL for the training set and validation set of the Places dataset used for augmentation on the DM-Control-GB benchmark.\n\n||Training set (1.8M Scenes)|Validation Set (36000 Scenes)|\n|-|----|---|\n|**Reward**|753.17|751.13|\n\nThis table implies that even with fewer types of natural images, the performance does not significantly degrade, implying that the diversity of natural images does not have a major impact on generalization performance. More effective types of images for augmentation would be those that resemble the distribution of the test environment. However, since we do not have prior information about the test environment, we used arbitrary natural images for augmentation and confirmed through various experiments that this approach is effective.\n\n***\n\n>**[Q2] Challenges anticipated in real-world scenarios**\n\n**(A)** A potential challenge when testing in real environments is the distribution shift, where the distribution of natural images used for augmentation during training differs from the distribution in the actual environment. However, our approach trains the model to focus on task-relevant objects while ignoring the background. As these task-relevant objects should consistently exist even in new environments with different backgrounds, we expect SimGRL to perform well in such cases. \n\n***\n\n>**[Q3] Perturbations that are not present in simulated benchmarks**\n\n**(A)** The robust performance of our method in DistractingCS in Appendix B.7, which includes perturbations such as camera pose distractions, suggests that our approach can handle types of disturbances not present in the training environment. We speculate that SimGRL’s robustness to distractions in this benchmark stems from its excellent ability to identify salient objects in the input, which helps in generalizing against distractions such as dynamic backgrounds or changes in camera pose.\n\n***\n\n>**[Q4] Effects of numbers of frames in the feature-level frame stack**\n\n**(A)** The number of stacked frames involves a trade-off between the amount of temporal information and the computational cost for encoding. If the number of stacked frames is reduced to less than three, the performance will degrade due to insufficient temporal information. On the other hand, since a feature-level frame stack encodes individual images separately in the image encoder, the number of encodings required increases in proportion to the number of frames, resulting in higher computational costs. Therefore, we adopted stacking three frames as in the previous methods and experimentally confirmed that this is the most appropriate frame stack.\n\n***\n\n>**[Q5] Sensitivity of the performance to the choice of hyperparameters**\n\n**(A)** As analyzed in Appendix B.3, the sensitivity to the number of layers in the image encoder was not highly critical. Even with just one layer, using a feature-level frame stack demonstrates significantly better generalization performance compared to an image-level frame stack. \n\nAdditionally, to show the impact of varying the maximum shift length (L) for the shifted random overlay, we present the test results on the \"Cartpole, Swingup\" task in 'Video Hard' for SimGRL-S, which uses only shifted random overlay augmentation without a feature-level frame stack.\n\n||L = 0|L = 3|L = 6|L = 9|\n|-|-|-|-|-|\n|**Reward**|393|713|724|701|\n\nIn this case, sensitivity to the maximum shift length is not very high, but there is a significant difference compared to the case of not using any shifting (L=0).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e7TaLtSsYw", "reviewer": "Reviewer_kgwp", "summary": "The paper first identify two pitfalls with the visual RL via gradient-based attribution mask, i.e., imbalanced saliency and observational overfitting. To address these two pitfalls, the paper proposes two novel modifications. One is for the encoder where encode each frame and stack encoded representations instead of encoding stacked frames directly. The other is a newly proposed shifted  randomly overlay augmentation method to address the observational overfitting.", "questions": "- Could you comment on whether the pitfalls also apply to robust RL algorithms like DBC, SLAC, TiA, DRIBO and RePo?\n- Could you comment on how different attribution methods may affect the identified pitfalls and TID metrics?", "limitations": "Yes, the author addressed the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- By and large, the paper is written well and presented clearly.\n- The section on pitfalls within visual RL algorithms is also interesting, and highlights the authors attention to the RL robustness and generalization problems.", "weaknesses": "- One concern I have about the paper is the first pitfall where the imbalanced saliency is specifically applied to visual RL algorithms that apply stacked frames. However, many robust RL algorithms, e.g., DBC, SLAC, TiA, DRIBO, RePo, take only single frame as the input to the encoder which apply similar ideas as the feature-level frame stack. These algorithms also explicitly take previous frame stack as input with an encoder $p_\\theta(s_t | s_{t-1}, a_{t-1}, o_t)$. It would be interesting to investigate whether these methods also suffer from the imbalanced saliency and compare the proposed feature-level stack method with them in terms of performance and efficiency.\n- Though the paper identifies the pitfalls with attribution methods, it would also be interesting to see attribution masks of SimGRL in different settings, i.e., clean, video easy and video hard. This would help validate that the identified pitfalls are sufficiently addressed by SimGRL.\n- Fig 5 and demos showed along the paper may be misleading that the policy is purely learned from the clean environment without any background perturbation. However, the proposed randomly shifted overlay augmentation explicitly introduces background perturbations into the training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your thoughtful comments. We address individual comments in the following.\n\n**[Weakness]**\n>**[W1] Comparison with robust RL algorithms**\n\n**(A)** We clarify that our work aims to address the generalization issue in a purely **model-free vision-based RL** setting, which requires the frame stack to encode temporal information. Due to the limitations of the rebuttal period, we present comparative results for DBC[1] and DRIBO[2] among the robust RL algorithms to investigate whether these methods also suffer from imbalanced saliency and compare them with the proposed method. \n\n+ DBC: Image-level frame stacking of 3 images, CNN encoder, Representation learning using a dynamics model, No strong data augmentation (e.g., Random overlay).\n\n+ DRIBO: No frame stack (using single images), Recurrent State-Space Model (RSSM) encoder, Representation learning using a dynamics model, No strong data augmentation (e.g., Random overlay).\n\n+ SimGRL (Ours): Feature-level frame stack of 3 images, CNN encoder, No representation learning and a dynamics model, Strong data augmentation (i.e., Shifted Random overlay).\n\nTo investigate whether these methods suffer from imbalanced saliency, we investigated TID metrics on 'Cartpole, Swingup', and compared them with SimGRL\n||DBC|\tDRIBO|\tSimGRL|\n|-|-|-|-|\n|**TID Score**|\t0.375|\t0.483|\t0.528|\n|**TID Variance**|\t284.5|\t33.7|\t22.8|\n\nThis table indicates that *1) DBC suffers from imbalanced saliency* (a lower TID Score and higher TID Variance) and *2) DRIBO does not suffer from imbalanced saliency* and can correctly recognize individual images (a higher TID Score and lower TID variance). Additionally, for better understanding, we provide the qualitative results of the masks and masked images for DBC and DRIBO in Fig. 1(a) of the attached PDF. These results imply that, when using an image-level frame stack as in DBC, imbalanced saliency persists even when the model-based RL setting is integrated. In contrast, when encoding single frames individually as in DRIBO, imbalanced saliency can be avoided, similar to feature-level frame stacking of the proposed method. \n\nHowever, our SimGRL has the advantage of being a simple yet effective structure, as it uses neither complex RNN-based networks like RSSM nor any additional representation learning strategies. In Figure 1(b) of the attached PDF, the training curves indicate the high sample efficiency in the training stage and superior test time performance of SimGRL. \n\n***\n\n>**[W2] Discussion on attribution masks in different settings**\n\n**(A)** In Section 5.2 and Appendix D.3, we have conducted an in-depth analysis of the pitfalls regarding clean videos and video hard scenarios. Additionally, in Appendix D.4, we have included rich example images of various attribution masks to demonstrate how effectively SimGRL addresses these pitfalls compared to existing methods. We would like to request you to refer to these sections. As for the 'Video Easy’ setting, it is much more generalizable than the 'Video Hard' setting, as it features simpler backgrounds and scene structures. Therefore, considering that the issues in the 'Video Hard’ setting are resolved, we assume that in the `Video Easy’ setting, which shows high test performance, the attribution mask is also well predicted, so we did not analyze it separately.\n\n***\n\n>**[W3]  Figure 5 that may be misleading**\n\n**(A)** For better clarity, we will add the augmentations used during training in Figure 5. We provide the modified figure in Figure 2(a) in the attached PDF.\n\n***\n\n**[Question]**\n\n>**[Q1] Whether the pitfalls also apply to robust RL algorithms**\n\n**(A)** As explained in **W1**, even robust RL algorithms can experience pitfalls when using image-level frame stacks, as seen with DBC, but these pitfalls will be resolved when sequentially embedding single images, as done in DRIBO.\n\n***\n\n>**[Q2] How different attribution methods may affect the identified pitfalls and TID metrics**\n\n**(A)** We clarify that the attribution masks are used solely for analysis and not for training, so they do not impact whether the RL models fall into pitfalls during training. Therefore, even if different methods are used to obtain attribution masks, as long as they correctly measure the influence of pixels in input images on the RL model, the identified pitfalls and TID metric values will not change significantly.\n\n ***\n[1] \"Learning invariant representations for reinforcement learning without reconstruction.\" ICLR (2021).\n\n[2] \"Dribo: Robust deep reinforcement learning via multi-view information bottleneck.\" ICML (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n46v7AMxgY", "reviewer": "Reviewer_wvnF", "summary": "This paper proposes two simple, yet crucial modifications to the Generalization in the Visual RL pipeline where during the test time, the visual observation consists of various degrees of dynamically varying backgrounds.\n\nConcretely, the authors identify two issues that inhibit RL agents from generalization: (i) imbalanced saliency and (ii) observational overfitting. To overcome these issues they propose two simple, yet effective solutions -- (a) Feature level frame stack where instead of stacking frames at the observation level, they stack at a feature level after obtaining spatial features of the images and (b) Shifted Random Overlay Augmentation where instead of applying data augmentation to the stack of frames, a shifted version of the data augmentation is applied to simulate the dynamically moving background effect which in turn helps learn more generalizable policy.\n\nThrough their experiments on the DM-Control (and robotic manipulation in the appendix) with varying backgrounds, the authors show the generalization capability of their proposed SimGRL (Simple framework for Generalization in visual RL) framework.  \n\nAdditionally, the authors also propose Task-Identification (TID) Metrics to evaluate how much of the policy can capture the task-relevant objects in each stacked frame.", "questions": "6.  Observational Overfitting: For this, one of the core arguments that authors provide is that because the same augmentation is applied uniformly to all frames the agent can tend to still focus on the background. I'm wondering how a baseline for this where you have different data augmentations for every frame in the history of stacked frames(say jittering, slight rotation, blur, overlay, random convolution, random shift) performs?\n\n7. In the Robotic manipulation results what does Test {1, 2, 3} mean? Does it mean running an evaluation on a _single episode_ for 5 seeds?\n\n----\n**References**\n\n[1] RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization, Zhecheng Yuan et al., NeurIPS 2023.\t\n\n[2] Deep Reinforcement Learning at the Edge of the Statistical Precipice, Rishabh Agarwal et al., NeurIPS 2021.\n\n----\n**Reason for current rating**: Because of my two major concerns around Attribute masking (and TID metrics), as well as lack of extensive experimentation on diverse RL environments, I am leaning towards a weak reject. However, my final decision would be based on authors' rebuttal, other reviewers comments and the discussion with authors during the rebuttal period. I'd encourage the authors to ask me any clarification questions for any of the experiments if they have any during the rebuttal/discussion period.\n\n-----\n\n**Post rebuttal update** -- As the authors have addressed all my major concerns, I have decided to increase my score to **Weak Accept**.", "limitations": "The authors don't mention any limitations of their work. It would be good to have a couple of lines highlighting where their method may fail. For instance, potentially large changes in background can lead to failure or their Feature stacking would work only in situations where Frame stacking is used -- model-based works like Dreamer do not use any frame stacking because of a recurrent network (such as GRU) that handles the history of states.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A key strength of this paper is the simplicity of the proposed solutions to address the generalization issue and these simple changes are quite effective as shown in the experiments.\n\n2. The intuition behind both the proposed solutions is clear and straightforward.", "weaknesses": "1. I do find myself in confusion with respect to the Attribution masking argument. I believe that the proposed solutions of Feature-level frame stack and Shifted Random overlay augmentation are both valid solutions to a generalizable policy irrespective of the authors inclusion of Attribution masking. My specific concern is as follows: \n\n    What is the guarantee that a clear thresholded gradient masking of the critic wrt the input, which consists of the complete segmentation of the relevant objects and agent in the mask will lead to a better generalizable policy? It can be possible that a generalizable policy would care about only a few key points that are predictive of the reward -- and hence be able to accurately predict the Value of a state, and in that sense, an accurate segmentation of the attribution mask need not be necessary. To give an example, in the robotic manipulation task of reach -- the reward function is typically shaped as some function of the Euclidean distance between the robot's end-effector and the goal position. In such a situation, the Value function shouldn't really be bothered about where the other parts of the robot are.\n\n2. The weakness in (1) ties into my second concern where if only a few parts of the agent are responsible for achieving (and predicting) reward, then I don't see the value of the proposed TID metric. This assumes that my entire object and agent in the scene is responsible for the policy, which as argued in (1) is not the case in most scenarios.\n\n3. [Relatively minor concern, not taken into account for the final scoring] It would be beneficial to the reader if the proposed components in Figure 4 could be highlighted with say a different color and a distinction between the existing pipeline of SVEA be made.\n\n4. On the experimentation end, I feel that the evaluation is very limited to DM-control tasks. While SimGRL performs quite well on Video-Easy and Video-Hard settings of DM-Control, and 2 other tasks of Robotic manipulation, I would have liked to see a more comprehensive evaluation of various visual RL environments. Specifically, I'd recommend performing evaluation on RL-ViGen [1] (https://gemcollector.github.io/RL-ViGen/)  which consists of a collection of diverse visual RL tasks. This would further strengthen this work's results.\n\n5. On L263, where authors say: \n> SimGRL demonstrates state-of-the-art performance in 4 out of 6 tasks in the Video Easy benchmark\n\n    One has to be careful while making such claims especially when the difference on one such task is +1 -- which in DM-Control doesn't really mean anything. I would ask the authors to look at IQM and Probability of Improvement metrics to compare SimGRL to other methods from [2] which has shown to be more reliable than taking an average over seeds. This does not involve re-running the experiments, the existing checkpoints and returns can be directly used over their wrapper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your thoughtful comments. We address individual comments in the following. Additionally, we sincerely appreciate your eagerness to engage in further discussions regarding our work!\n\n**[Weakness]**\n\n>**[W1] Clarity w.r.t the attribution masking argument**\n\n**(A)** *To avoid confusion, we clarify that our paper does not claim that an RL model must make a decision based on the ‘whole’ of task-relevant objects for better policy generalization.  Rather, we emphasize that it is important for the RL model not to be disturbed by task-irrelevant backgrounds.* To do this, we assumed that the RL agent should at least focus more on the task object than on the background, and employed the attribution masks to analyze whether the trained RL models correctly distinguish the task objects and backgrounds. Additionally, to quantitatively measure and analyze the ability to distinguish, we proposed the TID metric. We clarify that this metric is not designed to measure whether the models are making decisions based on the entire object.\n\nTo investigate whether the ability to distinguish between task objects and backgrounds also helps in identifying the key points that influence decisions, we provide attention maps obtained from the softmax of the attribution maps $M(Q_\\theta, s_t, a_t)$ before thresholding for the masks $M_\\rho(Q_\\theta, s_t, a_t)$ in the attached PDF. Figure 3 of the PDF shows that the identification ability for the task objects of SimGRL also leads to focus on the most important key point parts such as the pole and body of the 'Cartpole’ or legs of the 'Cheetah’ in the attention maps. On the other hand, the SVEA baseline incorrectly considers the background parts as key points. \n\n***\n\n>**[W2] TID Metric when a few parts of the agent are responsible for achieving reward**\n\n**(A)** As addressed in **W1**, we do not assume that the entire object is responsible for the policy's decision; instead, we assume it is important for the agent not to be disturbed by task-irrelevant backgrounds. Therefore, TID metrics are designed to measure how well the task-relevant object and background are distinguished, without considering which parts within the task-relevant object are more important.\n\n***\n\n>**[W3] Need for improvement of Figure 4**\n\n**(A)** We will improve Figure 4 by highlighting a distinction between the existing pipeline of the SVEA baseline. The modified figure is presented in Figure 2(b) in the PDF.\n\n***\n\n>**[W4] Additional experiments for RL-ViGen benchmark**\n\n**(A)** Due to the time constraints of the rebuttal period, we were only able to conduct a limited number of additional experiments on the RL-ViGen benchmark task.\nSpecifically, we present the experimental results on the 'Door' task in the Robosuite environment.\n\n|Task : Door|\tSVEA|\tSimGRL|\n|-|-|-|\n|**Train**|\t477.60|\t482.58|\n|**Test (Easy)**|\t112.86|\t200.51|\n\nThis table demonstrates that SimGRL achieves test performance gain against the baseline method in different simulation environments as well.\n\n***\n\n>**[W5] Measurement using IQM metric**\n\n**(A)** We evaluated the IQM (interquartile mean) scores, where we discarded the top and bottom values over the 5 seeds, and averaged the middle 3 seeds.\n\n||DMC-GB|SAC|RAD|DrQ|SODA|SVEA|TLDA|SGQN|SimGRL|\n|-|-|-|-|-|-|-|-|-|-|\n|**Video Easy**|Walker, Walk|262|540|741|\t846|\t888|\t847|\t895|\t**905**|\n||Walker, Stand|322|\t898|\t954|\t953|\t955|\t950|\t962|\t**966**|\n||Ball In Cup, Catch|\t134|\t432|\t495|\t658|\t922|\t868|\t869|\t**964**|\n||Cartpole, Swingup|\t348|\t478|\t428|\t705|\t779|\t695|\t642|\t**849**|\n||Finger, Spin|\t262|\t517|\t514|\t687|\t790|\t640|\t920|\t**982**|\n||Cheetah, Run|56|\t114|\t251|\t243|\t237|\t271|\t300|\t**311**|\n|**Video Hard**|Walker, Walk|\t71|\t54|\t88|\t378|\t532|\t424|\t726|\t**769**|\n||Walker, Stand\t|198|\t240|\t311|\t792|\t845|\t650|\t872|\t**934**|\n||Ball In Cup, Catch|\t85|\t73|\t131|\t116|\t514|\t472|\t835|\t**902**|\n||Cartpole, Swingup|\t112|\t135|\t120|\t410|\t405|\t320|\t383|\t**723**|\n||Finger, Spin|\t12|\t82|\t24|\t313|\t311|\t230|\t567|\t**868**|\n||Cheetah, Run|\t23|\t15|\t38|\t158|\t99|\t91|\t171|\t**299**|\n\nThe table shows that SimGRL also achieved robust performance in IQM evaluation.\n\n***\n\n**[Question]**\n\n>**[Q1] How does a baseline where we have different data augmentations for every frame in the history of stacked frames perform?**\n\n**(A)** To augment the background, it is better to apply the augmentation non-uniformly. However, if the augmentation directly affects the agent, using uniform augmentation would be more appropriate. For example, if random shifts are applied differently, the agent's position may change for each frame, which may distort temporal information. Therefore, the random shift (referred to as soft augmentation in the paper) applied by default, like in existing methods, was applied uniformly. On the other hand, we confirmed that shifting the random overlay, which has a greater influence on the background than the agent, is effective for generalizing to environments including dynamic perturbations. \n\n***\n\n>**[Q2] In the Robotic manipulation results what does Test {1, 2, 3} mean? Does it mean running an evaluation on a single episode for 5 seeds?**\n\n**(A)** Robot manipulation experiments were conducted by testing in each of the test environments—test 1, test 2, and test 3—after training in the training environment. Each test environment was evaluated with 30 episodes, and this process was repeated across a total of 5 different seeds for training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pa4yZXZ8Qk", "reviewer": "Reviewer_t66L", "summary": "This paper introduces SimGRL for vision-based DRL tasks. It tackles two challenges: the imbalanced saliency, where an agent repeatedly favors saliency maps from the recent one in stacked frames, and the issue of overfitting to particular background areas rather than concentrating on elements crucial to the task at hand. To overcome the first challenge, the proposed solution modifies the image encoding process, treating each frame separately before combining their features. To deal with the second issue of observational overfitting, this research uses an innovative method based on shifted random overlay augmentation. This method trains the agent to associate rewards with changes in the actual object of interest, thus learning to ignore background motions. Experiments were carried out using the DMControl-GB video benchmarks, evaluated by the TID score and variance metrics. The results show that SimGRL outperforms existing methods and the ablation study reveals each regularization technique independently contributes to notable enhancements.", "questions": "1) The findings concerning Video Easy presented in Table 1 appear to be of lesser importance. Could you provide a more detailed explanation of these findings?\n\n2) Furthermore, Table 1 reveals that 'Cartpole, Swingup' and 'Cheetah, Run' exhibit noteworthy outcomes, although the significance is not observed across all tasks. While you have offered explanations for the strong performance in 'Cartpole, Swingup' and 'Cheetah, Run', can you clarify why there weren't substantial improvements in some other tasks? Does this indicate that the impact of your study might be constrained?", "limitations": "No. The paper has limitations, but those are not discussed in the paper. The societal impact is not applicable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "In this study, the focus is set on addressing critical obstacles in DRL and the author suggested both straightforward and impactful alteration to the model's structure as well as enhancement in the augmentation techniques. The empirical outcomes are reasonable and the statistical findings detailed in Table 1 provide persuasive evidence of the effectiveness of their method.", "weaknesses": "The paper could benefit from an explicit articulation of its motivation and contributions. While multiple mentions are naturally made where the two challenges are being addressed, situating the motivation behind the work and a summary of contributions in a dedicated paragraph would enhance the strength of the paper. Although the devised SimGRL algorithm achieves top-tier performance on the Video Hard benchmark, the analysis of its performance on the Video Easy benchmark is not explored in depth, which presents an area for more detailed examination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your thoughtful comments. We address individual comments in the following.\n\n**[Weakness]**\n\n>**[W1] Need for an explicit articulation of its motivation and contributions**\n\n**(A)** We will add a paragraph to clarify the motivation and contribution of our paper. \n\n***\n\n>**[W2] Lack of in-depth analysis of the 'Video Easy' benchmark**\n\n**(A)** As for the `video easy’ benchmark, it is much more generalizable than the 'Video Hard' setting, as it features simpler backgrounds and scene structures. Therefore, given that the issues in video hard are resolved, we assume that video easy, which shows high test performance, is also addressed and did not analyze it separately. In addition, the performances for Video Easy of existing methods already are sufficiently saturated performance, and thus there is no significant performance difference between our method and existing methods. Therefore, we did not particularly emphasize the performance on Video Easy. The significance of the results for Video Easy in Table 1 is that our method demonstrates consistently excellent performance across both Video Easy and Video Hard environments, without being specifically favored to any particular environment.\n\n\n***\n\n**[Question]**\n\n>**[Q1] Need for a more detailed explanation of the results of 'Video Easy' in Table 1**\n\n**(A)** As discussed in **W2**, the significance of the results for Video Easy in Table 1 is that our method demonstrates consistently excellent performance across both Video Easy and Video Hard environments, without being specifically favored to any particular environment.\n\n***\n\n>**[Q2] Does the strong performance in 'Cartpole, Swingup' and 'Cheetah, Run' indicate that the impact of the study might be constrained?**\n\n**(A)** Existing methods show limited performance on 'Cartpole, Swingup' and 'Cheetah, Run' tasks compared to other relatively easy tasks where the performances are achieved to some extent. In contrast, our method provides consistently sufficient performance across all tasks, regardless of the type and difficulty of the task. Namely, while we see relatively greater performance improvements in the previously underperforming 'Cartpole, Swingup' and 'Cheetah, Run' tasks, we also achieve relatively small yet meaningful gains in other tasks. This is confirmed by the TID metric analysis for all tasks presented in Appendix D.3. In conclusion, we would like to emphasize that the impact of our research is not limited.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers, we appreciate all your valuable comments and the time you've dedicated to reviewing our paper.

Here, we briefly summarize the answers to some commonly asked questions and upload a PDF file with some additional figures to aid in understanding.

+ The attribution masks were used solely for identifying and analyzing the found pitfalls in existing model-free vision-based RL algorithms—imbalanced saliency and observational overfitting—and were not utilized during training.

+ Our main contributions are: 1) to the best of our knowledge, the first in-depth analysis of the pitfalls that hinder generalization in existing algorithms from the perspective of input frames, and 2) the demonstration that these problems can be alleviated and generalization significantly improved with a few simple modifications—namely, feature-level frame stacking and shifted random overlay augmentation.

+ We provide additional comparisons with methods that integrate model-based approaches, specifically DBC and DRIBO. The results are included in Figure 1 in the attached PDF.


Sincerely,

Authors

Meta-review: The reviewers advocate accepting this paper, and the AC agrees with this evaluation. During the rebuttal phase, the authors clarified the reviewers' confusion on the technical aspects and the experimental results. The reviewers found that the rebuttal adequately resolved the weaknesses. One minor note, the authors should update their references. Specifically, the authors cited the arXiv version instead of the conference version, e.g. [37] has been accepted to ICLR.
Decision: Accept (poster)