RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization
Authors: Zhecheng Yuan
Sizhe Yang
Pu Hua
Can Chang
Kaizhe Hu
Huazhe Xu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Visual Generalization
Visual RL
Benchmark
TL;DR: We introduce RL-ViGen: a novel Reinforcement Learning Benchmark for Visual Generalization, which contains diverse tasks and a wide spectrum of generalization types.

Abstract: Visual Reinforcement Learning (Visual RL), coupled with high-dimensional observations, has consistently confronted the long-standing challenge of out-of-distribution generalization. Despite the focus on algorithms aimed at resolving visual generalization problems, we argue that the devil is in the existing benchmarks as they are restricted to isolated tasks and generalization categories, undermining a comprehensive evaluation of agents' visual generalization capabilities. To bridge this gap, we introduce RL-ViGen: a novel **R**einforcement **L**earning Benchmark for **Vi**sual **Gen**eralization, which contains diverse tasks and a wide spectrum of generalization types, thereby facilitating the derivation of more reliable conclusions. Furthermore, RL-ViGen incorporates the latest generalization visual RL algorithms into a unified framework, under which the experiment results indicate that no single existing algorithm has prevailed universally across tasks. Our aspiration is that Rl-ViGen will serve as a catalyst in this area, and lay a foundation for the future creation of universal visual generalization RL agents suitable for real-world scenarios.  Access to our code and implemented algorithms is provided at https://gemcollector.github.io/RL-ViGen/.

Ri-e: A Reinforcement Learning Benchmark for Visual Generalization
Zhecheng Yuan 1,344 Sizhe Yang Pu Hua
Can Chang! 1.3,4
Kaizhe Hu Hu..3.4 Huazhe Xu 1,31
Tsinghua University, 2 University Electronic Science and Technology of China, Shanghai Qi Zhi Institute, *Shanghai Al Lab yuanzc230mails tsinghua. edu. cn, huazhe huazhe_xu@mail.
xuumail tsinghua, edu, cn
Abstract
Visual Reinforcement Learning (Visual RL), coupled with high-dimensional ob- servations, has consistently confronted the long-standing challenge of out-of- distribution generalization. Despite the focus on algorithms aimed at resolving visual generalization problems, we argue that the devil the existing benchmarks as they are restricted to isolated tasks and generalization categories, undermining comprehensive evaluation of agents' visual generalization capabilities. To bridge this gap, we introduce RL-ViGen: novel Reinforcement Learning Benchmark for Visual Generalization, which contains diverse tasks and a wide spectrum generalization types, thereby facilitating the derivation of more reliable conclu- sions. Furthermore, RL-ViGen incorporates the latest generalization visual RL algorithms into unified framework, under which the experiment results indicate that no single existing algorithm has prevailed universally across tasks. Our aspi- ration is that RL-ViGen will serve catalyst in this area, and lay a foundation for the future creation of universal visual generalization RL agents suitable for real-world scenarios. Access to code and implemented algorithms provided at
git.hbitthuuu io/ilo/RRl RL- ViCen/
Introduction
Visual Reinforcement Learning (RL) has attained remarkable success across a plethora of do-
mains 39 43 14]. A diverse range of techniques has been implemented to tackle not only the
tria--and-error learning process but also the complexity arising from nigh-dimensional input data. Notwithstanding these successes, fundamental challenge confronting visual RL agents persists achieving generalization.
To overcome this obstacle, several visual RL generalization benchmarks have emerged, including Procgen (6) Distracting Control Suite 47 and DMC-GB (21]. While these benchmarks have been indispensable to visual RL generalization progress, they are not exempt from inherent limitations that pose challenges to further devclopment. Procgen offers diverse distribution of environment configurations and visual appearances. However, it is limited vidco games with non-realistic images and oow-dimensional discrete action spaces, resulting in a significant gap between its environments and real-world scenarios. Another instance, DMC-GB, is sometimes treated golden standard for many state-of-the-ar visual generalization algorithms.
equal contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Training
Generalized Visual Scenarios
Table-top Manipulation
Autonomous Driving
Dexterous
Manipulation
Locomotion
Indoor
Navigation
CH
Figure 1: The novel RL benchmark for visual generalization. We show that RL-ViGen supports wide range of tasks with different generalization categories. The algorithms can be evaluated more comprehensively and achieve more convincing experimental results.
Nevertheless, the narrow scope of task classes and generalization categories in existing setups cannot thoroughly and comprehensively evaluate the agent's generalization ability. In addition, although Distracting Control Suite contains
generalization types, falls short in diversity and complexity. The prevailing trend in this field is to showcase the superiority of proposed algorithms on these benchmarks, which adversely poses certain risk of promoting overfitting to these benchmarks, rather than discovering algorithms potentially beneficial for solving real-world problems.
In this paper, we introduce novel Reinforcement Learning benchmark for Visual Generalization (RL- ViGen), presenting numerous merits over existing counterparts. Our benchmark integrates spec- trum of task categories with realistic image inputs, including table-top manipulation, locomotion, autonomous driving, indoor navigation, and dexterous hand manipulation, allowing for more com- prehensive evaluation of the agents' efficacy. Moreover, by incorporating various key aspects in visual RL generalization, such as visual appearances, lighting changes, camera views, scene structures, and cross embodiments, RL-ViGer enables a comprehensive examination of agents' generalization ability against distinct visual conditions.
It is noteworthy that we provide a unified framework that encompasses various state-of-the-art visual RL and generalization algorithms with the same optimization scheme each approach. The framework not only promotes fair benchmarking comparisons but also lowers the entry barrier for devising novel approaches.
In summary, our contributions are as follows: 1) we propose a novel visual RL generalization benchmark RL-ViGen with diverse, realistic rendering tasks and numerous generalization types; implement and evaluate various algorithms within unified framework, enabling comprehensive analysis of their generalization performance; we conduct comprehensive and extensive experiments to demonstrate the distinct performance of existing approaches when tackling diverse tasks and generalization types, and highlight the benefits and the limitations of current generalizable visual RL algorithms. With all the contributions combined, RL-ViGes may pave the way for further advancements in visual RI. generalization, ultimately leading to more robust and adaptable algorithms for real-world applications.

---

Table 1: Generalization Categories. The following table outlines the types of generalization incorporated within each task. Except for categories considered as not applicable (N/A) (e.g., for locomotion, changes in scene structures are not required), all potential types are included.
Generalization, Categories
Visual Appearances
Camera
Lighting Changes
Scene Structures Embodiments
Cross
Views
Autonomous Driving
Table-top Manipulation
V
V
N/A
Indoor Navigation
N/A
Dexterous Manipulation
N/A
Locomotion
N/A
2 RL-ViGen
RL-ViGen consists of 5 distinct task categories, spanning the domain of locomotion, table-top manipulation, autonomous driving, indoor navigation, and dexterous hand manipulation. In contrast to prior benchmarks, RL-ViGen employs a diverse array of task classes for evaluating the agent's generalization performance. We believe that only through comprehensive examination from multiple perspectives can we obtain convincing results. Furthermore, as shown in Table our benchmark offers a wide range of generalization categories, including visual appearances, camera views, variations in lighting conditions, scene structures, and cross embodiments settings, thereby providing thorough evaluation of algorithms' robustness and generalization abilities.
2.1 Environments
Dexterous manipulation: Adroit [44] is sophisticated environment that is explicitly tailored for dexterous hand manipulation tasks. It demands considerable exploration and fine-grained feature capturing due to the sparse reward nature the environment and the complexity of high-dimensional action space. In RL-ViGen we have enriched the Adroit environment by integrating diverse visual appearances, camera perspectives, hand types, lighting changes, and object shapes.
Autonomous driving: CARLA 9 serves as a realistic and high-fidelity simulator for autonomous driving, which investigates the control capabilities of agents under dynamic conditions. has been successfully deployed on visual RL settings in prior studies. Contrary to previous work [24). RL- ViGen provides an enhanced range of dynamic weather and more complex road conditions different scene structures. Furthermore, flexible camera angle adjustments are also included within RI-VViGen.
Indoor navigation: As an efficient and photorealistic 3D simulator, Habitat [46] combines numerous visual navigation tasks. Succeeding in these tasks requires the agents to own the capability of scene understanding. RL-ViGen builds upon the skokloster-cal castle scene and proposes additional scenarios with different visual and lighting settings. In addition, the camera view and scene structure are designed to be adjustable.
Table-top manipulation: Robosuite (64) is a modular simulation platform designed to support robot learning. It inherently contains interfaces designed to adjust various scene parameters. Recent work 12 has leveraged this platform to test the agent's generalization ability of visual background changes. RL-ViGen further incorporates dynamic backgrounds, adaptive lighting conditions, and options for embodiment variation, refining the simulation to be closer to the real world.
DeepMind Control is popular continuous visual RL benchmark. DMC-GB 22 is developed on and has become a widely used benchmark for evaluating generalization algorithms. Building upon DMC-GB, RL- RL-Vi introduces objects and corresponding tasks from sophisticated real-world locomotion and manipulation applications, such as the Unitree, Anymal quadrupedal robots, and the Franka Arm. What's more, RL-ViGen also offers variety of generalization categories to further enrich this environment.
Locomotion:
More detailed implementations and modifications be found in Appendix and our codebase.

---

Stage 1: Learning generalizable agent
Actor
Action
Critic
Observation
Encoder
Representation
Value
Stage 2: Generalize scenarios
Execute
Action
Figure Generalization procedure. The agent is first trained in Stage with a certain fixed scenario. Subsequently, in Stage 2, the agent is tested across various visual generalization scenes in a zero-shot manner. The better the agent performs various scenes of Stage 2, the stronger generalization ability it demonstrates.
2.2 Generalization Categories
Here, emphasize the primary generalization categories utilized in RL-ViGen
Maintaining effective performance in the presence of altered visual features of objects, scenes, or environments is of vital importance, particularly for visual reinforcement learning. In our benchmark, different components within the environment can be modified with wide range of colors. Meanwhile, the dynamic video background is also introduced as a challenging setting.
Visual appearances:
Camera views: In the real world, the agents have cope with camera configurations, angles, or positions that may not align with those experienced during training. We offer access to set the cameras different angles, distances, and FOVs. In addition, the number of cameras can be adjusted accordingly.
Lighting conditions:
The change in lighting conditions will occur inevitably in the real world. To equip agents with the ability to adapt to such variations, our benchmark supplies interfaces related to the lighting, such varied light intensity, colors, and dynamical shadow changes.
Scene structures: Mastering the ability understanding and adapting to different spatial arrange- ments and organization patterns within various scenes crucial for truly generalizable agent. To this end, our benchmark enables modifications in scene structure via adjusting maps, patterns, or introducing extra objects.
Cross embodiments: Adapting learned skills and knowledge to different physical morphologies or embodiments is essential for an agent to perform well across various platforms or robots with different kinematic structures and sensor configurations. Therefore, our benchmark also provides access to modify the embodiment of trained agents in the aspects of model types, sizes, and other physical properties.
Algorithmic Baselines for Generalization in Visual RL
3.1 A Unified Framework
Another key contribution our work is the implementation unified codebase to support compar- ison among various visual RL algorithms. In previous studies, different algorithms adopt distinct optimization schemes, RI. baselines, and hyperparameters. For example, SRM (25) and SVEA [33
rely on SAC-based RL algorithms, while PIE-G [6]] utilizes DDPG-base based approach. Morcover, mi-

---

nor different implementations could substantially impact the final performance. Therefore, providing a unified framework is of great importance in this domain, enabling more persuasive conclusions to be drawn from evaluating algorithms across consistent framework and diverse tasks.
3.2 Visual RL Algorithms
In our benchmark, we assemble eight leading visual RL algorithms and apply the same unified training and evaluation framework. DrQ-v2 [57] is the prior state-of of-the-art DDPG-basee model-free visual RL algorithm in terms of sample efficiency. DrQ [32] is another SAC-based sample efficient visual RL algorithm, which is the base of DrQ-v2. CURL 3 utilizes SimCI R-sylee [5] contrastive loss to obtain better visual representations. VRL33 50 is the state-of-the-ar algorithm in Adroit tasks with human demonstrations. The other four algorithms concentrate on achieving robust representations. SVEA 23 employs the Q-value of un-augmented images as the target objective while utilizing data augmentation for reducing the Q-variance; SRM (25] adopts augmentation in the frequency domain to selectively eliminate a part of the observation frequency; PIE-G [6] incorporates ImageNet [8] pre- traieed model to further boost the generalization ability; SGQN 3 identifies critical pixels for Secision-making via integrating with the saliency map.
Experiments
In this section, we try to investigate the generalization ability of different approaches in the proposed RL-ViGen benchmark. As shown in Figure 2, all agents are trained in the same fixed training environment and evaluated within various unseen scenarios in a zero-shot manner. The training sample efficiency and asymptotic performance are shown in Appendix F.4 For each task, we evaluate 5ver random seeds and report the mean scores and 95% confidence intervals. In terms of each trained environment, we present the aggregated scores of the multiple subtasks. The detailed and extensive experimental results can be found in Appendix and F The visualization of each environment and generalization types are shown in Appendix D.I
4.1 Visual Appearances and Lighting Changes
4.1.1
Indoor Navigation
Within the Habitat platform, we choose the Im- ageNav task and modify the 3D scanned models to introduce novel scenarios with various visual appearances and lighting conditions. We con- duct 10 evaluations in each of the 10 selected scenarios (100 trials in total). In contrast to most existing benchmarks, the Habitat-rendered im- ages are captured from first- firs-peesoon viewpoint by the high-performance 3D simulator. Hence, it can deliver visualization more akin real- world scenes. As shown in Figure 3 the superior performance of PIE-G can be attributed to the in- tegration with the ImageNet pre-trained model, equipping PIE-G with wealth of authentic im- ages and enabling it to handle these scenarios more efficiently. Conversely, consistent with the conelusion drawn from Section 4.1.2, SGQN, which intends to segment the centered agent via eliminating the redundant background, is proved ineffective in these object-rich and first- rest-person view tasks.
Habitat Generalization
Success Rate (9)
70
SYEA
(x0+9
DVO
24
15
Figure 3: Generalization score of indoor naviga- tion. We present the success rate of each method. The result indicates that PIE-G achieves better gen- eralization performance on Habitat.
4.1.2 Autonomous Driving
Regarding CARLA, we adopt the reward function setting in Zhang et al. [62 and apply first-person perspective to better resemble real-world driving conditions. As shown in Figure I in Appendix [D.]] this environment is divided into three levels: Easy, Medium, and Hard. The main modifications involve varying factors such as rainfall intensity, road wetness, and lighting. The higher the disparity from the training scenarios, the more challenging the difficulty level. In this task, one of its distinctive
5

---

features is that the input image frequency undergoes considerable changes. Consequently, the SRM approach, which applies data augmentation the frequency domain, demonstrates the best performance as it can adapt to the input images with varying frequencies. While PIE-G incorporates the ImageNet pre-trained model, its source training images mainly possess higher-frequeney features, thus suffering from suboptimal generalization when facing low-frequency scenarios (e.g., dark night). Moreover, SGQN, which extracts salient information, exhibits a decrease in performance when faced with visually rich scenes where the controlled agent is not present in the observed frame. It also should be noted that DrQ gains a degree of generalization ability in this environment. Our observations suggest that since DrQ is SAC-based algorithm, it tends to be prone entropy collapse [57]. This implies that the trained agent only produces a single distribution of action in response to diverse image inputs.
CARLA EASY
CARLA MEDIUM
CARLA HARD
Episode Return
57
65
45
34
38
32
30
22
22
22
8,7
3.2
1.
SGQN
PIE-G
SRM
SVEA
DrQ-v2
DrQ
CURL
Figure 4: Aggregated generalization score of autonomous driving, We present the aggregated return of each method. SRM exhibits better performance adapt scenarios where image frequency varies dramatically.
4.1.3 Dexterous Hand Manipulation
In the Adroit environment, we assess the performance of each approach in three single-view tasks: Door, Hammer, and Pen. Since DrQ-v2 and DrQ barely perform well in these challenging environ- ments, we utilize VRL3 [50], the statc-of-the-art method in this domain, as the base algorithm and the visual RL approaches in RL-ViGen are re-implemented upon it.
With respect sample efficiency, it is commonly believed that applying strong augmentation will negatively affect sample efficiency. However, as illustrated in Figure 5 in Appendix F.4 it
worth noting that since VRL3 specifically designs a safe Q mechanism prevent potential Q divergence for this environment, the generalization algorithms applying strong augmentation can achieve performance comparable to those using only random shift.
door
pen
hammer
1
Success Rate
0
SGQN
PIE-G
SRM
SVEA
VRL3
CURL
Figure 5: Sample efficiency of Adroit. The success rate of each algorithm. We normalize the training
steps into (0, 1). The approaches with strong augmentation can also gain comparable performance.
As for generalization, Adroit tasks require agents to identify fine-grained features for dexterous and sophisticated manipulation. Therefore, PIE-G, which leverages ImageNet pre- mre-traine models

---

Color-Easy
Color-Hard
Video-Ease
Video-Hard
62
65
66 66
Success Rate (%)
59
60
50
56
42
32
37
34
34
32
2,3
20
0.00_5
0.00.6
0.00.5
1,20,3
SGQN
PIE-G
SRM
SVEA
VRL3
CURL
Figure The aggregated generalization score of dexterous manipulation. We present the aggregated success rate of each method. PIE-G equipped with the ImageNet pre-trained model exhibits better adaptability Adroit tasks which necessitate fine-grained information capture.
to capture detailed information, demonstrates the effectiveness of assisting the learned agent in executing downstream tasks, particularly in the hard setting. Moreover, as illustrated in Figure 6 the absence of additional objectives to mitigate the effect of visual changes causes both VRL3 and CURL to struggle adapting to novel visual situations these demanding tasks.
4.2 Scene Structures
Generalizable agents that are capable of delivering robust performance across diverse scene structures essential for potential broad real- worldworl applica- tions. select CARLA as the testbed for evaluating the generalization of scene structures. The agents are trained in standard training scenarios (highways), and tested in more complex structure settings, in- cluding narrow roads, tunnels, and roundabouts with HardRainSunsel weather conditions. As shown in Figure 7 the performance of all algorithms falls short of expectations, suggesting that the current visual RL algorithms and generalization approaches are not adequately robust scene structural changes. More in-depth investigations must be pursued in order to enhance the generalization ability of trained agents to perceive the changing scene structures.
Scene Structure Generalization Score
13
Episode Return
-0.1
-2.5
SGON
DrO CURL
PIE-G
SVEA
Figure 7: Generalization score of Scene Structure. Across this category of general- ization, algorithms demonstrate unsatisfac- tory performance.
4.3 Camera Views
We proceed to evaluate the generalization in terms of camera views
Camera EASY
Camara HARD
21
Success Rate (%)
53
in the Adroit Environ- ment. As illustrated in Figure/ [ under the Easy setting, PIE- and SGQN exhibit leading generalization capabilities with re- spect to camera view, while other algorithms also demonstrate some degree of generalization due to the use of random shift augmentation. However, in the Hard setting, which introduces substantial changes in camera position, orientation, and field of view (FOV), nearly all algorithms, except for SGQN, lose their generalization ability. The exceptional performance of SGQN is mainly due to its heavy reliance producing saliency maps, which enhances the agent's
42
22
23
13
9.9
SGON
PIE-G
SRM
SVEA
VRL3
CURL
Figure 8: Generalization score of Camera Views. SGQN indicates ad- vantageous generalization ability across different levels in camera-view generalization.
7

---

self-awarenesss of object geometry and relative positioning. Hence, this property strengthens its generalization performance even the face of major camera view alterations.
4.4 Cross Embodiments
Addressing the embodiment mismatch from visual input is crucial, as the embodi- ment composes substantial portion of the image and significantly influences robot be- havior of interacting with the world. To investigate this type of generalization, Ro- bosuite is employed the evaluation plat- form. We utilize the OSC_ POSE con- troller 41 during training facilitate the maintenance of action space dimensions and their respective meanings. Then, the trained agents transfer from Panda Arm to two different morphologies: KUKA IIWA and Kinova3. As illustrated in Figure the overall performance of all algorithms is suboptimal; however, generalization-bu based methods, which contain more diverse information during training, exhibit a slight advantage over those primarily focused on sample efficiency in the cross-embodiment setting.
Cross Embodiment Generalization Score
Episade Return
103
103
98
91
SCON
PIE
SVEA
CURL
Figure 9: The aggregated generalization score of Cross Embodiments. No algorithm has demonstrated the capability to manage the cross-embodiment general- ization yet.
9
5 Discussion
In summary, our experiments reveal that the findings based previous benchmarks may not accu- rately reflect the actual progress, leading to distorted perception the situation; those advanced visual RL algorithms, previously perceived as cutting-edge, display less efficacy within RL-ViGen. We summarize the main takeaways as follows:
Takeaway 1.
The experimental results reveal the varying generalization performance of differ- ent visual RL algorithms distinct tasks and generalization categories, with no single algorithm demonstrating universally strong generalization abilities.
Takeaway 2. Solely enhancing training performance fails guarantee improvement in the generalization ability. Although DrQ(v2) and CURL exhibit high sample efficiency during training and even achieve better asymptotic performance (Appendix F.4), their performance in various generalization scenes has yet to reach a satisfactory level. Therefore, when attempting to improve the generalization ability of agent, is crucial to introduce additional inductive biases aid the training process.
Takeaway 3. An effective generalizable visual RL agent must demonstrate exceptional performance across multiple generalization categories. Previous work has primarily focused on generalization concerning visual appearances, while our experiments reveal considerable shorteomings of exist- ing algorithms in the setting of cross embodiments and scene structures. These underperforming generalization categories go beyond altering the observation space within the Markov Decision Process (MDP); they also bring modifications to the action space and transition probabilities, thus presenting the agent with extra challenges.
Takeaway Each generalization algorithm possesses its own unique strengths. Notably, PIE-G demonstrates superior performance with respect to visual appearances and lighting condition changes, while SRM, under significant image frequency variations, exhibits remarkable robustness. SGQN retains its generalization capacity when facing considerable camera view alterations. In addition, SVEA, without the need for additional parameters and with only minimal modifications, can achieve certain level of generalization abilities. We hypothesize that stronger performance might be attained through a fusion of different algorithms, such as utilizing pre-trained models with rreueeny-basee augmentation to induce further improvement.
Combined with the takeaways, we hope that an algorithm's 's success RL- can indicate its potential applicability in more complex and unpredictable real-world scenarios. In the future,

---

holistic and cailti-dimesionnal approach, encompassing aspects such as scene structures, camera views, and cross embodiments, is critical for fostering truly generalizable agents capable of navigating in varied and dynamic real-world environments. Equally, the design of more sophisticated and realistic training environments which enable to reflect the complexity of real-world conditions, can also serve as crucial area for future explorations.
6 Related Work
RL benchmarks. There exists a multitude of mature benchmarks aiming for evaluating reinforce-
ment learning algorithms 4 [1 38 10
40 55]. For instance, Atari (2) and Gym-MuJoCo [48]
are exemplary benchmarks in deep reinforcement learning. In other subdomains, D4RL. [15 serves as a popular benchmark for offline RL algorithms, while URLB [34] provides evaluating platform with respect to unsupervised RL algorithms. MetaWorld (59] is often used to evaluate multi-task and meta-learning scenarios. SafetyGym [45), meanwhile, is predominantly applied for testing Safe RL algorithms. Recently, MineDojo [3 benchmarks embodied agents in exploration and multi-task domains. Contrasting to these benchmarks, RL- RL-ViGen distinguishes itself by incorporating a variety of task classes and an array of generalization categories and primarily focuses on evaluating agents' visual generalization abilities.
Generalization. How to endow models' generalization abilities is a pivotal topic in machine learning. In computer vision, well-established benchmarks are available for exploring generalization problems [54] [53 58 31]. While several approaches have been proposed in RL and robotics to tackle
30, 28 (66 60), the benchmarks in use are relatively immature [[] 6. 47 [2 fraught with numerous limitations, and lacking unified framework for comparison.
such issues 23
25
For example, Procgen 6 widely used benchmark for quantifying the agents' generalization abilities. However, Procgen remains video game platform, with the human-imaged world rather real-world counterparts, offering limited assistance for agents' generalization in real-world scenarios. The Distracting Control Suite [47 and DMC-GB [21]. building upon DM-Control 49 introduce some types of visual distractions. Nevertheless, their tasks are solely focused on locomotion, and there remains a substantial gap between this simulated environment and real-world scenarios. By contrast, RL- ViGen encompasses various forms of generalization, exhibits a high degree of photo- realism, and includes diverse range of tasks. Avalon (1] is another valuable benehmark for RL generalization. It shares a unified world dynamics and task structure, making it highly suitable as a benchmark for in-distribution generalization. Contrary to Avalon, which is mainly concerned with task-level generalization, RL-ViGen mainly focuses on diitrr-iistiiittionn generalization, with a specific concentration on the visual aspects of generalization.
7 Conclusion, Limitations, and Future Work
In this work, we propose novel Reinforcement Learning benchmark for Visual Generalization (RL- ViGen), a comprehensive benchmark for evaluating the visual generalization abilities of trained agents. RL-ViGen stands apart from existing benchmarks by boasting broader diversity of tasks and generalization categories, which in turn fosters more persuasive conclusions. According to the quantitative experimental results from RL-ViGen, we note that, of now, there are existing generalization algorithms that can adeptly manage all tasks and generalization types. is our expectation that the advent of RL-ViGen will bring fresh perspectives to the research community, and stimulate the advancement of agents that can truly exhibit overall visual generalization capabilities.
Limitations, The agents trained through RL-ViGen have not yet been evaluated in real-world scenarios. In our future work, we would like build certain real-world tasks to demonstrate the value that RL-ViGen can provide in developing generalizable agents for real-world applications.
8 Acknowledgements
This work is supported by research program 20222D0161700.

---

References
[1] Joshua Albrecht, Abraham Fetterman, Bryden Fogelman, Ellie Kitanidis, Bartosz Wróblewski, Nicole Seo, Michael Rosenthal, Maksis Knutins, Zack Polizzi, James Simon, et al. Avalon: A benchmark for rl generalization using procedurally generated worlds. Advances in Neural Information Processing Systems, 15:12813-12825 2022.
[2] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-211 279, 2013.
[3] David Bertoin, Adil Zouitine, Mehdi Zouitine, and Emmanuel Rachelson. Look where you look! saiiency-guided g-networks for generalization in visual reinforcement learning. Advances in Neural Information Processing Systems, 15:30693-30700 2022.
[4] Siwei Chen, Yiqing Xu, Cunjun Yu, Linfeng Li, Xiao Ma, Zhongwen Xu, and David Hsu. Daxbench: Benchmarking deformable object manipulation with differentiable physics. In The Eleventh International Conference on Learning Representations, 2023.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-16077 PMI.R, 2020.
[6] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048-204 2048-2056. PMLR, 2020.
[7] MuJoCo Menageric Contributors. MuJoCo Menageric: A collection of high-quality simulation models for MuloCo, 2022. URL http::////giii com http///eeemnnnmdddeoede cotooooooeaaeeiieeee
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[9] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. Conference on robot learning, pages 1-16. PMLR, 2017.
[10] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, PMLR, 2016.
pages
[11] Mohamed Elsayed, Kimia Hassanzadeh, Nhat M Nguyen, Montgomery Alban, Xiru Zhu, Daniel Graves, and Jun Luo. Ultra: A reinforcement learning generalization benchmark for autonomous driving, 2020.
[12] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5088-3099 PMLR, 18-24 Jul 2021.
[13] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open- embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL attpss//opeereviie www.:eeeeeee.ooorreerrrrooooeee
[14] Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615 (7953):600-627,2))) 2023.
[15] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.07211, 2020.
10

---

[16] Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar. Nick Haber, al. Threedworid: platform for interactive multi-modal physical simulation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). 2021.
[17] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiaing Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In International Conference Learning Representations, 2023.
[18] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: a large-scale dataset of minecraft demonstrations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2442-2448, 2019.
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[20] Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2021.
[21] Nicklas Hansen and Xiaolong Wang- Generalization reinforcement learning by soft data augmentation. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 13611-13611 IEEE, 2021.
[22] Nicklas Hansen and Xiaolong Wang. Generalization reinforcement learning by soft data augmentation. In International Conference on Robotics and Automation, 2021.
[23] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.
[24] Yangru Huang Haoran Xu, Ding Chen. Carlaenv-benchmark,, 2022.
[25] Yangru Huang, Peixi Peng, Yifan Zhao, Guangyao Chen, and Yonghong Tian. Spectrum random masking for generalization in image-based reinforcement learning. Advances in Neural Information Processing Systems, 35:20393-20406, 2022.
[26] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation.
[27] Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: generalization challenge in vision, control, and planning. arXiv preprint arXiv: 1902.01378. 2019.
[28] Kibeom Kim, Min Whoo Lee, Yoonsung Kim, JeHwan Ryu, Minsu Lee, and Byoung- Tak Zhang. Goal-aware cross-entropy for multi-target reinforcement learning. Advances Neural Information Processing Systems, 34:2783 2795, 2021.
[29] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:2111 1.09994, 2021.
[30] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 201-264, 2023.
[31] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in-the-wila distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[32] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforeement learning from pixels. arXiv preprint arXiv:2004.13649) 2020.
11

---

[33] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[34] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Picter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:21 15191, 2021.
[35] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33:19884 19895, 2020.
[36] Xiang Li, Jinghuan Shang, Srijan Das, and Michael Ryoo. Does self-supervised learning really improve reinforcement learning from pixels? Advances in Neural Information Processing Systems, 1-308865-3888) 2022.
[37] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34, 2021.
[38] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Piere Berges, Pieter Abbeel, Jitendra Malik, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303. 18240, 2023.
[39] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7550):529-553 5333, 2015.
[40] Tongzhou Mu, Zhan Ling, Fanbo Xiang. Derek Cathera Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[41] Jun Nakanishi, Rick Cory, Michael Mistry, Jan Peters, and Stefan Schaal. Operational space control: A theoretical and empirical comparison. International Journal of Robotics Research, 27(6):737-757, 2008
[42] Steve Nebel, Sascha Schneider, and Günter Daniel Rey. Mining learning and crafting scientific experiments: a literature review on the use of minecraft in education and research. Journal of Educational Technology & Society, 19(2):355-366 2016.
[43] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[44] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: /70iv pat.::..990.0070 10087, 2017.
[45] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. Openai Blog, 2019.
[46] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[47] Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting con- trol suite-a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.0272. 2021.
12

---

[48] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basel control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5033- IEFE, 2012. doi: 10.1 10.1009/ROSS200 20w........0...6.66600
[49] Saran Tunyauuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. _om_otrool: Software and tasks for continuous control. Softwrre Impacts, 6: 100022,, 2020.
[50] Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vrl3: A data-driven framework for visual deep reinforcement learning. In Conference on Neural Information Processing Systems, 2022. URL
[51] Tingwu Wang, Xuchan Bao, Igmasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv: arXiv:1907.02057, 2019.
[52] Hua Wei, Jingxiao Chen, Xiyang Ji, Hongyang Qin, Minwen Deng, Siqin Li, Liang Wang, Weinan Zhang, Yong Yu, Liu Linc, al. Honor of kings arena: an environment for generalization in competitive reinforcement learning. Advances in Neural Information Processing Systems, 35:
1 11881-11892. 2022.
[53] Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of- distribution generalization in transfer learning. Advances in Neural Information Processing Systems, 35:7181-7199 2022.
[54] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Dvijotham, and Taylan Cemgil. A fine-grained analysis on distribution shift. arXiv preprint 1.Xi::2100.11328 2021.
Ira Ktena, Krishnamurthy
[55] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[56] Zhengyu Yang, Kan Ren, Xufang Luo, Minghuan Liu, Weiqing Liu, Jiang Bian, Weinan Zhang, and Dongsheng Li. Towards applicable reinforcement learning: Improving the generalization and sample efficiency with policy ensemble. arXiv preprint arXiv:2205.09284, 2022.
[57] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data- reinforcement learning. arXiv preprint arXiv:2107.09645. 2021.
[58] Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions of out-of distrribitii generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7947-7958, 2022.
[59] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1 1100. PMLR, 2020.
[60] Zhecheng Yuan, Guozheng Ma, Yao Mu, Bo Xia, Bo Yuan, Xueqian Wang, Ping Luo, and Huazhe Xu. Don't touch what matters: Task-aware lipschitz data augmentation for visual 2022.
reinforcement learning. arXiv preprint
[61] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. arXiv preprint arXiv: arXiv:2212.08886, 2022.
[62] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint 1077:2006..0742, 2020.
13

---

[63] Tony Z. Zhao, Siddharth Karamcheti, Kollar Thomas, Chelsea Finn, and Liang Percy, What makes representation learning from videos hard for control? https:/honychaazh.ghhhhi wtt.://wwaaaoo....uu..oooooo io/ 2023.
[64] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu, robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint wrXi::2009.12293, 2020.
14

---

Appendix
A Visual Reinforcement Learning Baselines
DrQ:
This model-free, off-policy reinforcement learning algorithm, is based on Soft Actor-Critic (SAC) [19]. DrQ enhances training stability via applying data augmentation regularize the Q value of state-action pairs. The key of DrQ is to promote similarity between augmented state-action pairs. The Q-regularization technique is shown Eq where K the number of samples, T the collection of augmentation.
E s-pp()) [Q(s, a)] = (f ax) where Me E T and ag ~ π (-1ffs",,m)) (I) am3)(11))
K
1=1
DrQ-v2: An improved version of DrQ. DrQ-v2 fuses essential elements from the DDPG algorithm with data augmentation to strengthen visual RL agents' performance. DrQ-v2 also incorporates techniques such as n-step return and target critic, leading to commendable results in most of the medium and hard level DM-Control tasks. The TD-target is defined as follows, where Iz+n is the n-step observation, at+n is the n-step action, and 01,2 is the Q-target networks:
m
? Tee + 7dh min Qa, (aug(Detn), at+n) k=1,2
CURL integrates contrastive learning methods into the reinforcement learning training process. The auxiliary contrastive loss (Eq allows the agent obtain better image representation during training, thus mitigating the optimization difficulty under high-dimensional inputs. In our implementation, we only apply a single encoder to produce visual representations instead of two polyak-averaging encoders. This alteration improves the sample efficiency of CURL and put it comparable performance with DrQ-v2. More experiments are shown in Appendix F3
CURL:
exp (/Wk) K-1
Lgg log exp WWh4) E exp (Kk)
PIE-G proposes a simple yet effective method, combining ImagNet pre-trained visual representations with the early layer and updates of BatchNorm statistical parameters to further enhance the generalization ability of the agent.
PIE-G:
SVEA finds that heavy data augmentation introduces additional high variance to agent training, which can lead to instability or even divergence. SVEA suggests that using the Q-values of non-augmented images as the target of estimated Q-values for augmented images (Eq 4), thus stabilizing the variance the value estimation.
SVEA:
||Qo (aug(ax)) ax) tgt
(4)
SRM proposes novel data augmentation method that operates in the frequency domain. It helps diversify data and alleviate distribution shift issues under various visual scenarios. During the training stage, SRM randomly discards parts of the frequency information from observations, forcing the policy to select suitable actions based on the remaining information. The augmentation method is shown in Eq where F is the fast Fourier transform, M is binary masking matrix, and Z is random noise image.
SRM:
=(o)) + + (1-1) F(Z)
(5)
SGQN:
This algorithm introduces the saliency map for the use of augmenting images. Saliency maps, tool used in computer vision, offers interpretability analysis of encoders. SGQN retains only agent's ss focusing areas and removes the visual background by the generated saliency map.
15

---

This approach utilizes the augmentation objectives in SVEA 23 to further improve the model's generalization performance. The auxiliary objective is shown in Eq where Mp((o, a), a) is the binary masking matrix introduced from the saliency map.
Lc |Q0 (o, - Qo (0 M,(o.a),a
(6)
B Implementation Details
Indoor navigation: Habitat serves as the simulator and extends variety of indoor navigation tasks. We select ImageNav the test env, whose goal is defined by the image of target location in the chosen map. Due to the complexity the default training and validation episode settings, which demands extensive training periods to achieve satisfactory standard, we simplify the setup to 500 initial positions and target position. Meanwhile, we utilize the 3D scenes from the Gibson dataset our map for all experiments.
Autonomous driving: We choose the stable version of CARLA 0.9.10 for simulation. The reward function is adopted from Zhang et al. [62]. We also implement the wrapping methods from Huang al. [24) for novel CARLA environments. Moreover, to enhance exploration and ensure stable training, standardized the std schedule across all algorithms. Each difficulty level contains two weathers, Easy level: soft high light, soft noisy_ low light; Medium level: HardRainSunset. SoftRainSunset, Hard level: hard_ low hard_lowllight, hard lod__noss_lovv low_light. The aggregated return is calculated by averaging over the weather at the same level. Further details can be accessed the documentation provided within GitHub repository.
Dexterous manipulation: In RL-ViGen, we select three single-view Adroit tasks. Given that tasks in Adroit typically necessitate demonstrations for successful completion, employ VRL3, the state-of-the-art baseline for these tasks. Since the update process of VRL3 is based on DrQ-v2, it allows a seamless transfer of our algorithms VRL3 training: stagel responses to gain a basic perception ability via pretraining on ImageNet; stage2 utilizes offline RL training with expert demonstrations; stage3 executes online training. is noteworthy to mention that the experiment of VRL3 demonstrates that in single view tasks, only applying stage3 is sufficient to accomplish Adroit tasks with high sample efficiency. Therefore, to compare each algorithm more effectively, we exclude the use of stagel and stage2. The aggregated success rate calculated by averaging over all three tasks.
VRL3's codebase. There are three stages for
Table-top manipulation: SECANT 12 previously employed Robosuite for generalization testing.
Building on its codebase, we adopt one of the latest versions Robosuite 1 1.4.0 and mujoco 2.3.0 as well as simplified the installation process. Mcanwhile, we also introduce range of new classes of visual generalization. For each difficulty level, we deploy a variety of scenarios, and each trained agent is evaluated within each environment 10 times (in a total of 100 evaluations). The aggregated return is calculated by averaging over all three tasks.
Locomotion: In addition to the locomotion tasks from DM-Control (1.0.8 version), we also incorporate models from Mujocoreic [7], and carefully designed corresponding rewards, enabling them accomplish walk or stand tasks. Furthermore, building DMC-GB, we have added additional generalization categories for further enriching RL-ViGen. The aggregated return is calculated by averaging two tasks.
Our experiments are conducted with TeslaA40 or TeslaA 100 GPU and AMD EPYC 7542 32-Core Processor CPU. More details can be founded in thwb. htwm///gmcolleecc///LLLL RRt.lleeettoooooooeeeea- RL
C
Hyper-parameters
We use the same achieve better performance of certain algorithms. The common hyper-parameters listed in Table 2
as the original papers and perform a small-scale grid search to
The individual byper-parameters are listed the following Tables. The additional hyper-parameters introduced by SGQN are listed well.
16

---

Table 2: Common hyper- ppprrpparamters in RL- Vien.
Hyper-parameters
Value
Input size
84 x 84
Discount factor y
0.99
Replay Buffer size
int(le7) DrQ(v2), CURL: 50, otherwise: 256
Feature dim
Action repeat
Robosuite: 1, otherwise: 2
N-step return
DrQ: I, otherwise: 3
Optimizer
Adam 1024
Hidden dim
Frame stack
3
Table 3: CARLA hyper-parameters in RL-ViGen.
Hyper-paramettrs
Value
Training Frames
int( (e66)
Learning Rate
PIE-G: 5e-5, DrQ: 5e-4, otherwise: le-4
N-step return
SGQN quantile
0.9
SGQN critic weight
0.5
SGQN aux Ir
8c-5
D
Visualization of each difficulty level
To gain better understanding of our setting and RL-ViGen, we visualize the images under various generalization settings and difficulty levels mentioned the experiment section.
D.1
Visual Appearances and Lighting Changes
D.1.1 Robosuite
In the context of Robosuite, each difficulty level comprises 10 distinct scenes. We perform 10 trials for each of these scenes (100 trials in total).
The Easy level includes changes of the background appearance, while the Hard level contains additional complexities of moving light and alterations the robotic arm's color. The Extreme level further employs dynamic video background to evaluate the trained agents' generalization abilities. The visualized figures are shown Figure 10
Training
Easy
Hard
Extreme
Figure 10: The visualization of various difficulty levels of Robosuite, This figure shows examples from the Door task. As the difficulty level increases, more types of distracting factors are introduced.
D.1.2 CARLA
Apart from the default weather settings in CARLA, we implement series of challenging new scenarios. In our CARLA setup, each level of difficulty characterized by two specific weather conditions. The Easy level includes soft fiiiioooiy___l low light and high_ligh while the Medium
17

---

Table 4: Habitat hyper-parameters in RL- RL.ViGen.
Hyper-parametee
Value
Training Frames
int(7e5)
Learning Rate
le-4
N-step return
1
SGQN quantile
0.93
SGQN critic weight
0.9
SGQN aux Ir
8e-5
Table 5: Adroit hyper-parameters in RL-VViGen.
Hyper-parameters
Task
Value
Hammer
int( 1c6)
Training Frames
Door
int(1e6)
Pen
int(2e6)
Hammer
le-4
Learning Rate
Door
le-4
Pen
le-4
Hammer
0.9
SGQN quantile
Door
0.9
Pen
0.9
Hammer
0.9
SGQN critic weight
Door
0.5
Pen
0.9
Hammer
8e-5
SGQN aux Ir
Door
8e-5
Pen
8e-5
level is defined by the HardRainSunset and SoftRainSunser conditions. The Hard level contains hard low light and hard noisy low light. As the disparity between the novel scenarios and the training images increases, the difficulty level of generalization grows. RL-ViGen also encompasses challenging conditions such as rainy, overcast, and slippery road surfaces. The visualized figures are shown in Figure
Easy
Medium
Hard
Figure 11: The visualization of various difficult level of CARLA. The higher the disparity from the training observations, the more challenging the new scenario.
D.1.3 Habitat
For Habitat, the Gestaltor 3D model editor applied modify the appearance of the scene's 3D models. A total of 10 distinct scenarios created. The visualized figures are shown in Figure 12
18

---

Table Robosuite hyper-parameters in RL-V RL-ViGen.
Hyper-parameters
Task
Value
Door
int(6e5)
Training Frames
Lift
int(8e5)
TwwAroPellnhhoc
int(8e5)
Door
le-4
Learning Rate
Lift
DrQ(v2), CURL: Ie-4, otherwise: 8e-5
TwoArmPeglnhole
SGQN: le-4, otherise: 8c-5
Door
Easy Medium Medium
Level
Lift
TwoArePeglnhooe
Door
0.9
SGQN quantile
Lift
0.9
TwoArePegInholc
0.87
Door
0.7
SGQN critic weight
Lift
0.7
TwoAroPeelnholc
0.7
Door
8e-5
SGQN aux Ir
Lift
8e-5
TwoAroPeginhooe
8e-5
Figure 12: The visualization of Habitat. We create 10 distinct scenarios for the generalization of visual appearances.
D.1.4 Locomotion
For DM-Control, we further augment numerous new types of generalizations on the basis of DMC- GB. For the unitree series tasks, Easy and Hard denote two levels of difficulty regarding light color, light position, changes of light's movement and objects' color. The visualized figures are shown in Figure
13
D.1.5 Adroit
In the Adroit environment, we provide four generalization scenarios. The Color setting changes the background, object color, and table texture, while the Video setting utilizes dynamic background and introduces moving light. As illustrated Figure 14 each scenario configured with two levels of difficulty.
D.2 Camera Views
For camera view generalization, we implement alternations to the camera view through the modifica- tion of camera's 's orientation, position, and FOV. The visualized figure is shown in Figure [5
19

---

Train
Easy
Hard
Figure 13: The visualization of various difficulty level of DM-Control. The figure above show examples from unitree tasks. Factors such as light color, light position, movement of light, and object color are varied.
Train
Color-Easy
Color-Hard
Video-Easy
Video-Hard
Figure 14: The visualization of various difficulty levels of Adroit. This figure demonstrates examples from the Pen task. We show four generalization scenarios provided in RL- ViGnn....
D.3 Scene Structures
As shown in Figure 16 established variety of road scenarios in CARLA, including roundabouts, narrow paths, tunnels, etc., which can be also utilized conjunction with other adjustable parameters. As the experiment illustrated in Figure 7 we employ the same weather conditions those during training.
D.4 Cross Embodiments
In terms of cross-embodiment generalization, we modify the type of the robotic arm in Robosuite. In addtion, by leveraging the OSC_ POSE control method, the input actions interpreted delta values from the current state, thus facilitating to maintain the action space dimensions and corresponding meanings.
E Visualization of each difficulty level
To gain better understanding of our setting and RL-ViGen, we visualize the images under various generalization settings and difficulty levels mentioned in the experiment section.
At first, we will structure the general modifications made to each generalization type. A more detailed configuration of each environment will be specified subsequently.
Visual appearances: The generalization type of visual appearances are divided into two variations: static and dynamic changes.
Color: Drawing on DMC-GB [22), we categorize color variations into two levels (easy and hard) difficulty, each containing 100 color combinations. In both levels, alter- ations are made to three environmental attributes: body, grid, and skybox. In the easy level, the contrast between the color combinations and the original training scene's color combinations is 0.07, while in the hard level, the contrast increases 0. 14.
Video: We divide video variations into two levels difficulty by replacing the envi- ronment's skybox with vidcos. The easy level contains 10 videos, while the hard level
20

---

Original
Easy
Hard
Figure 15: The visualization of camera views of Adroit. The larger the deviation angle of the camera, the higher the difficulty of generalization.
Roundabout
Narrow
Tunnel
Figure 16: The visualization of scene structures of CARLA, We selected certain locations within different maps to as scenarios for scene structure generalization.
consists of 100 videos with increased contrast. Additionally, the hard level also remove the reference plane of the ground.
Lighting conditions: The generalization type of lighting conditions are divided into two variations: static and dynamic changes. Each variations are categorized into two difficulty levels.
Static: We adjust the static lighting conditions by altering the position, intensity, and color.
* Position: Easy: the coordinates z,y will be determined by adding values randomly sampled from the interval [-0.5, 0.5] via uniform sampling to the original light source coordinates, and the z coordinate will be selected from the range [-0.2, 0.2]; Hard: the coordinates z.y will be determined by adding values randomly sampled from the interval [-1, 1] via uniform sampling to the original light source coordinates, and the coordinate will be selected from the range [-0.5, 0.5).
Intensity: Easy: the value of intensity will be uniformly sampled from the interval [0.7, 1.4]; Easy: the value of intensity will be uniformly sampled from the interval [1.4, 2.5].
Color: Easy: the RGB values will be determined by adding values randomly sampled from the interval 1-0.1, 0.1] via uniform sampling to each channel; Hard: the RGB values will be determined by adding values randomly sampled from the interval [-0.2, 0.2] via uniform sampling to each channel. It should be noted that the RGB values normalized [0, 1].
Dynamic: The alteration y coordinate is defined by adding a value within the interval [-1.2, 1.2] the original coordinate value, with incremental change of 0.04 each step.
Camera views: The generalization type of camera views divided into two variations: static and dynamic changes. Each variations are categorized into two difficulty levels.
Position: Easy: the coordinates æ,y,z will be determined by adding values randomly sampled from the interval [-0.03, 0.03] via uniform sampling to the original camera positions; Hard: the coordinates z,y.e will be determined by adding values randomly
21

---

Panda
IIWA
Kinova3
Figure 17: The visualization of cross embodiment of Robosuite. This figure shows examples from the Door task. Here, we demonstrate our modification of the style of the robotic arm for cross-embodiment generalization.
sampled from the interval [-0.07, 0.07] via uniform sampling to the original camera positions.
Orientation: Easy: each component of quaternion will determined by adding values randomly sampled from the interval [-0.03, 0.03] via uniform sampling to the original camera quaternion; Hand: each component of quaternion will determined by adding values randomly sampled from the interval [-0.08, 0.08] via uniform sampling the original camera quaternion.
FOV: Easy: the coordinates 2.y.2 will be determined by adding values randomly sampled from the interval [-5, 5] via uniform sampling to the original FOV; Hard: the coordinates zyy. will be determined by adding values randomly sampled from the interval [-10, 10] via uniform sampling to the original FOV.
Due to the challenges in defining the difficulties associated with the generalization types of the scene structures and cross embodiments, and the complexity in numerically describing the differences between scenes, don't provide additional quantification to these two types.
F
Additional Results
F.
Generalization Evaluation
E.1.1
Locomotion
Built upon DM-Control, which has included numerous locomotion tasks, we extend this benchmark by integrating real-world robot models from Mujocoreie 7 with corresponding tasks. Moreover, RL-ViGen also augments DMC-GB with more tasks and generalization types. Here we evaluate the performance of each algorithm on the Unitree series tasks. Figure [8] demonstrates that all generalization algorithms exhibit comparable performance. More specifically, SVEA outperforms other techniques in the Easy setting, where the other generalization techniques do not show any advantages. In the Hard setting, where the agent's color closely resembles that of the surrounding environment, SGQN may not effectively capture the agent's outline, leading to performance decline.
F.1.2 Table-top Manipulation
In Robosuite, three tasks, including single-arm and dual-arm settings, are selected in RL-ViGen: Door, Lift, and TwoArmPegInhole.. Additionally, we create multiple difficulty levels, incorporating various visual scenarios, and dynamic backgrounds. In the Easy and Medium test environments, where considerable variations in visual colors and lighting changes are introduced, the results in Figure 19/sow that PIE-G demonstrates slightly better performance than that of SGQN and SRM in Easy and Medium settings. However, when faced with the Hard setting that integrates dynamic video backgrounds, SRM, which mainly resorts static Trequency-based augmentation, is unable to adapt effectively to such scenarios for completing the manipulation tasks. Figure further indicates that the remaining algorithms struggle to demonstrate generalization abilities in this environment.
22

---

Locomotion EASY
Locomotion HARD
332
343
Episode Return
260
326
236
253
224
202
204
201
131
80
53
SGQN
PIE-G
SRM
SVEA
DrQ-v2
CURL
DrQ
Figure 18: Generalization score of Locomotion. The generalization algorithms show comparable performance at two difficulty levels.
MEDIUM
HARD
305275
257
Episode Return
146
156144
142142
4,8
59
52
50
64 T
50
SGQN
PIE-G
SRM
SVEA
DrQ-v2
DrQ
CURL
Figure 19:
The aggregated generalization score of table-top manipulation. We present the aggregated return of three tasks for each method. PIE-G shows better generalization performance of table-top manipulation tasks when facing unseen visual scenarios.
F.2
Wall Time
So far, our main focus has been the comparison of general- ization performance of each method across various tasks. In this section, we turn our attention to the comparison of each algorithm's wall-clock training time. We choose Walker walk task from DMControl for evaluation. This task requires a large batch size for training, thus suit- able for better demonstrating the wall-time efficiency of each approach. Frames-per-secone the evaluation metric. Figure 20 illustrates that DrQ-v2 owns the least computational cost. Conversely, for the al- gorithms that utilize additional data for augmentation pur- poses, they tend to exhibit lower trames-per-secome (FPS) rates. SGQN builds the saliency maps during every train- ing step, which takes extra costs. Meanwhile, PIE-G uti- lizes the ImageNet tre-trdined ResNet model to convert high-dimensionnl images into representations, thus adding more burden on the model's inference compared to other algorithms.
Wall Time Comparison
75
44
(FPS) is selected to be
33
35
16
SCON PEF-G
SRM
DrQ+2
DrD
SVIA
CuRi
Figure 20: Wall Time Comparison.
DrQ-v2 enjoys the lowest computational cost.
F.3 The re-implementation of CURL
which adopts contrastive loss as an auxiliary objective, is frequently mentioned in numer- ous works 57 32 35). yet the effectiveness of contrastive loss appears to be less pronounced [35] 36]. Distinct from prior studies, we not utilize target encoder and remove the update momentum parameters related to the encoder. As shown in Figure 21 comparing to the state-of-the-an of-the-art approach DrQ-v2 and the results reported in previous work [57]. the use of single shared encoder for achieving representations seems to yield more favorable results when leveraging contrastive loss.
CURL
33
23

---

walker walk
hopper hop
quadruped
1000
Episode Reward
800
Epiode Reward
400
DrQ-v2
CURL
800
700
DrQ-v2
CURL
300
600
500
200
400
300
200
DrQ-v2
100
200
CURL
0-
100 10
300 10
300 x 10*
quadruped
finger spin
1000
700
Epicode Reward
800
600
Episode Reward
600
500
600
400
400
Eplsod
300
400
200
DrQ-v2
200
CURL
100
CURL
CURL
0
0-
300
300
10
100 10*
Frame
Frame
Frame
Figure 21: The sample efficiency comparison between CURL and DrQ-v2. Our remiplemenntnn of CURL can achieve comparable sample efficiency with DrQ-v2.
Door
TwoArmPegInHole
Lift
500
Episode Return
0-
Figure Sample efficiency of Robosuite. The episode return each algorithm. We normalize the training steps into (0. 1). DrQ-v2 and CURL show better sample efficiency.
F44
Sample efficiency
In this section, compare the sample efficiency of various visual RL algorithms. As one of the state- of-the-art visual RL algorithms, DrQ-v2 serves as baseline for evaluating the training performance of various algorithms across different tasks. In each figure, the convergence performance of DrQ-v2 marked with a gray dashed line.As shown Figure 22 22] and Figure 23 DrQ-v2 and CURL obtain advantageous sample efficiency in locomotion and table-top manipulation tasks. A shared attribute between these two types of tasks is that the agent is positioned at the center of observation. Hence, the additional noise introduced by data augmentation tends exacerbate training instability.
Regarding Habitat and CARLA, as shown in Figure 24 the difference of sample efficiency across diverse algorithms is minimal. This may be attributed to the fact that both two environments employ first-person view rendered images, which makes them more robust to the extra noise. Besides, should be noted that in CARLA, agents required to execute fast action changes the roads to avert collisions with surrounding vehicles. Therefore, Figure 24b demonstrates that DrQ is prone to entropy collapse, while SGQN struggles extract salient information with many distracted factors.
In terms of Adroit, as mentioned in Section 4.1.3 the safe Q mechanism able to endow the trained agent with robustness against noise. The sample efficiency of each algorithm is shown in Figure

---

unitree, stand
unitrre, walk
Eskoodo huturn
650
420
FBG
5006
50M
5000
SVEA
200
0-0
DrQ
DIO
-200
400 10
700 20'
Frames
Fromess
(b) Figure 23: Sample efficiency Unitree tasks. The episode return of each method. The agent, positioned at the center of observation in these tasks, is subjected to additional noise due to data augmentation.
(a)
Hobitat Training
CARLA Training Curve
1.0
100
SOON
PE-G
0.8-
SRM
SVE4
Episade Return
0-g
0r0
0.6
CLRI
0.4
07
0.-
10
20
30
40
300
Frame (x30f)
Frame [x30%)
(a)
(b)
Figure 24: Sample efficiency of Habitat and CARLA. We show the success rate of Habitat and the episode return CARLA accordingly. The first-person view observations are more robust to augmentation of adding additional noise.
F55
Training on the full-distribution scenarios
To investigate the performance of agents under test and full-distribution scenarios, we select three locomotion tasks: walker walk, finger spin, and walker stand. In terms of algorithmic choices, employ three algorithms, DrQ-v2, SRM, and SVEA, to validate the training and generalization efficacy.
We have devised three distinct visual scenarios, denoted as Scenario 1, Scenario 2, and Scenario for training agents and evaluating their generalization performance. Scenario which serves the training environment within the Section 4 is constructed in a static and uncluttered visual setting. Scenario employed the testing scenario for visual appearances in work, introduces dynamic complexity by integrating video backgrounds. Scenario 3, characterized as a full distribution scenario, further amplifies this complexity by incorporating additional visual generalization types, such as changes in camera view and lighting conditions. The visualized figure are shown Figure 25
First, We explore the training performance of different algorithms across various visual scenarios. As shown in Figure 27 it demonstrates that as the distribution expands and the incorporated variations increase, noticeable decline is observed in both the sample efficiency and asymptotic performance across all algorithms.
Subsequently, we investigate the generalization performance of each trained agent across three visual environments. During the generalization testing, we conduct evaluations a zero-shot manner. The complete generalization performance are shown in Figure 28 With the increasing complexity of the testing scenarios, the generalization scores tend to decline. Moreover, the generalization scores are directly correlated with the training performance; despite the fact that Scenario 3 incorporates the
25

---

Scenarlo
Scenario
Scenario 3
Walker Walk
Finger Spin
Walker Stand
Figure 25: Visualization of three trained tasks in different visual scenarios. The scenario is constructed in a static and clean visual scenario. Conversely, the scenario introduces dynamic variations by employing video background that alters with each episode, encompassing total of 110 distinct videos. The scenario further extends this complexity by incorporating additional visual generalization types, such as changes in camera view and lighting conditions.
Aggregated Generalization Performance
Lighting Conditions
80
Scene Structures
Camera Views
Visual
Cross effbodiments
Appearances
CURL
DrQ-v2
Figure 26: Aggregated score of each algorithm. We show the aggregated combine metric for each agent. There no single algorithm can lead generalization types.
most extensive array of generalization types, the inferior training performance of the agent leads to suboptimal generalization performance across various scenarios.
F.6
The aggregated score of each algorithm
Additionally, we present an aggregated metric for each algorithm across different types of general- ization. We normalize the score of each environment into 0-100, and then take the average score across all environments for each generalization type. As shown in Figure [26 no single visual RL algorithm has proven capable adeptly handling types of generalization, especially in the types of scene structures and cross embodiments settings. In the Discussion section, we have analyse the underlying causes of these challenges. It provides direction for guiding future efforts towards secking approaches that can achieve further generalization advancements in these two aspects.
26

---

G
Additional Related Work
Honor of Kings Arena [52 serves as another benchmark for RL generalization. Distinct from RL-
ViGen, it functions as multi-agent platform, focusing mainly on the generalization of targets and opponents rather than visual aspects. While platforms like MineRL [18] and Malmo [26]. built on
Minecraft [42], are capable of handling a variety of tasks, the construction of these tasks tends to be relatively simplistic without fine-grained modeling of agents and objects. Crafter [20] and the
Obstacle Tower [27]. on the other hand, still utilize discrete actions, and the task types they offer are limited and lack diversity. The benchmarks such as BEHAVIOR [37] and ThreeDWorld [T6] present photo-realistic environments, but their task visual scenarios also relatively narrow and are not applicable for visual generalization evaluation.
H
Additional Discussion
The augmentations during training. Both DrQ and DrQ-v2 employ data augmentation techniques like random shift or random crop. Such augmentations are referred to as weak augmentations, which only introduce minor changes the image such slight cropping and shifting. Numerous studies [35 57 32 have shown that weak augmentations are indispensable for image-based RL achieve high sample efficiency. Absence of such weak augmentations could easily fail most tasks [35]. However, when it comes to generalization, weak augmentations cannot help agents to obtain generalization abilities [35]. On the other hand, the augmentation method utilizing extra datasets falls into the category of strong augmentations [23] [2],, which substantially distorts the image. Contrary to weak augmentations, this approach is imperative to foster superior generalization capabilities, but it will hinder agent's training performance. Therefore, most generalization a algorithms utilize both types of augmentations sample efficiency.
to achieve generalization ability while maintaining high
drqv2
svea sconaric1
srm
1000.0
1000.0
sconarol 01
sconari02
sconaria 02
sconorio2
Sconar92
800.0-
800.0
sceneenc3
Egisode Reward
Episede Reward
600.0
600.0
600.0
Egisode
200.0
400.0
200.0
200.0
0.0
0.0
0.0
100 10
Frame
Frame
Figure 27: Training Curves of three algorithms in different visual scenarios. The legends are defined to represent various training curves under different scenarios. As the distribution expands and the incorporated variations increase, a noticeable decline observed in both the sample efficiency and asymptotic performance across all algorithms.
27

---

Generalization Score on Scenario
Generalization Score on Scenario
Generalization Score on Scenario
Farccs os sccnancl
france en scenaa
francd soma100
Faiied scenaic?
Trakned soma
Takeed coma
959
955
Episade Return
041
743
747
527
4.5
ego
262
227
287
245
206
163
221
74.65
100
1024
DiQ+22
SVEA
SRM
DrQ-v2
SVEA
SRM
DrQ+v2
SVEA
SRM
(b)
(0)
Figure 28: Generalization performance on different visual scenarios. this figure, the general- ization performance of each agent is evaluated across three types of visual scenarios. Specifically, the blue, yellow, and green bars represent the generalization scores evaluated in a certain scene for the agents trained under Scenariol, Scenario2 and Scenario3, Each bar represents the average performance across three tasks.
28


Reviews:
[{"review_id": "sIimbsU5eX", "reviewer": "Reviewer_jky4", "summary": "", "questions": "", "limitations": "This is a good paper. Some minor issues should be addressed.\n\n1. In line 163, “This implies that the trained agent only produces a single distribution of action in response to diverse image inputs.” Have you verified this?\n\n2. DrQ and DrQ-v2 use data augmentation techniques to do representation learning. These data augmentation approaches overlap to some extent with the augmented dataset approach used in this paper. Therefore an additional discussion of this point will help the reader to understand this benchmark.\n\n3. The new benchmark as a whole falls under the category of presenting new challenges to the generalizability of reinforcement learning from the **visual side**. How does this new benchmark benefit the RL algorithms themselves rather than how to do representation learning better?\n\nI would be happy to raise my score if these minor issues were fixed.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper discusses the generalization of deep RL, including building a new diverse benchmark and evaluating related deep RL algorithms, which is a very important topic for the RL community. This may inspire a new series of studies on the generalizability of reinforcement learning.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "7VvoWJhaBJ", "reviewer": "Reviewer_Jh47", "summary": "", "questions": "", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ A very comprehensive benchmark on visual RL generalization with a wide range of tasks that are very relevant to robotics learning and autonomous driving;\n+ The visual generalization types are in good coverage;\n+ The benchmarked algorithms are state-of-the-art and the latest developments in the direction;\n+ All benchmarked algorithms are (re-)implemented in a unified codebase where they could make fair comparisons across algorithms;\n+ They concluded that no single algorithm has demonstrated winning performance across all evaluated tasks; also each latest algorithm in the benchmarks has its own unique strengths; ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "O1jFiyooz5", "reviewer": "Reviewer_jzXK", "summary": "", "questions": "", "limitations": "1. I agree that cross embodiment is an important category to guide future work, especially in the long term, but I question whether it is in the right topic range as a visual RL benchmark for generalization.\n2. Overall, the study is good for experimenting in different settings, but needs to reconsider whether the experiments were appropriate for each task.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Research on reinforcement learning to process vision input, which is a high-dimensional observation, is a time-consuming work, and especially reinforcement learning that is parameter sensitive is not easy to learn successfully. Under these circumstances, I believe that experiments on various tasks and their generalization would be beneficial to the RL community.\n2. As written in the above contributions, I believe that the variety of generalization types and tasks is a key strength of this study.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "nurHA9NlOI", "reviewer": "Reviewer_WGQU", "summary": "", "questions": "", "limitations": "- re-implementing RL algorithms correctly is very hard - introducing subtle mistakes or differences to the original is common and can cause differences in performance. before i feel comfortable using a new implementation of an algorithm, or trusting experimental results, i like to see proof that the re-implementation matches the original implementation. this is usually best done by running the old and new implementations on the same environments and showing they match, or you can also show that your implementation matches results from the original papers.\n- each of these individual environments is not super high quality. combining a bunch of sorta crappy environments together into one benchmark feels like it just sorta masks over the issues. if you're working with one environment, you can deeply understand the limitations and work around them. but when you're using a suite like this, you kinda just treat the environments as a black box.\n- in general, i'd prefer approaches that make a single high-quality environment that can encompass the various types of generalization all within the same environment. eg i am one of the authors of https://github.com/Avalon-Benchmark/avalon , which (obviously i am biased) i feel like could have been adapted easily to make an out-of-distribution-generalization benchmark (we focused on in-distribution generalization) that was more understandable and robust than the one presented here.\n- installing 5 separate RL environments seems like it would be a pain from a usability perspective", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- it seems that there's not a good complete benchmark for this subfield (trusting the authors didn't omit anything important in the related work), so i think they are addressing a useful problem\n- implementing multiple algorithms in a unified framework is hard, and definitely can help facilitate research if the implementations are high-quality and proven to accurately reproduce the original results.\n- the experimental results may be interesting to those trying to understand what the current state of the art is in this subfield\n- having multiple different types of environment in one benchmark can help see the strengths and weaknesses of each algorithm", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper has received the scores 6, 7, and 8, with all reviewers unanimously voting for acceptance. They have highlighted the paper's significant contribution to advancing research in the generalization of deep reinforcement learning (RL) with high-dimensional visual inputs. Given the potential impact and the novel contributions of this study, I recommend acceptance.

Additionally, I went through the code available in the GitHub repository, and I encountered several issues that raised questions about the benchmark's reliability. It would be great if the following issues can be addressed before final submission:

1. CARLA Evaluation: The testing revealed that the weather configuration in the CARLA evaluation does not consistently function as intended. It remains uncertain whether the root of this issue lies with the CARLA simulator or the benchmark code in the repository. This inconsistency has the potential to influence the experimental results materially, necessitating a comprehensive resolution to ensure the application of accurate configurations.
2. Discrepancy in SVEA Algorithm Performance: I observed a notable discrepancy in the performance of the SVEA algorithm within the CARLA-easy environment, which registered a score of 56, markedly lower than the score cited in the paper. This deviation could potentially be influenced by the aforementioned issue and underscores the necessity to address this variance to uphold the findings' integrity.
3. Provision of Raw Scores: To record the stratified bootstrapped confidence interval for each algorithm, it is vital to provide the raw scores for each seed instead of simply offering aggregated scores in the GitHub repository.

I remain confident that, by diligently addressing these concerns, the paper can firmly establish itself as a robust contribution to the field.
Decision: Accept (Poster)