SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhecheng Yuan $^{1,3,4*}$ , Sizhe Yang $^{2,3*}$ , Pu Hua $^{1,3}$ , Can Chang $^{1,3,4}$ , Kaizhe Hu $^{1,3,4}$ , Huazhe Xu $^{1,3,4}$
<sup>1</sup> Tsinghua University, <sup>2</sup> University of Electronic Science and Technology of China, <sup>3</sup> Shanghai Qi Zhi Institute, <sup>4</sup> Shanghai AI Lab yuanzc23@mails.tsinghua.edu.cn, huazhe_xu@mail.tsinghua.edu.cn
Visual Reinforcement Learning (Visual RL), coupled with high-dimensional observations, has consistently confronted the long-standing challenge of out-of-distribution generalization. Despite the focus on algorithms aimed at resolving visual generalization problems, we argue that the devil is in the existing benchmarks as they are restricted to isolated tasks and generalization categories, undermining a comprehensive evaluation of agents' visual generalization capabilities. To bridge this gap, we introduce RL-ViGen: a novel Reinforcement Learning Benchmark for Visual Generalization, which contains diverse tasks and a wide spectrum of generalization types, thereby facilitating the derivation of more reliable conclusions. Furthermore, RL-ViGen incorporates the latest generalization visual RL algorithms into a unified framework, under which the experiment results indicate that no single existing algorithm has prevailed universally across tasks. Our aspiration is that RL-ViGen will serve as a catalyst in this area, and lay a foundation for the future creation of universal visual generalization RL agents suitable for real-world scenarios. Access to our code and implemented algorithms is provided at https://gemcollector.github.io/RL-ViGen/.
Visual Reinforcement Learning (RL) has attained remarkable success across a plethora of domains [39, 43, 14]. A diverse range of techniques has been implemented to tackle not only the trial-and-error learning process but also the complexity arising from high-dimensional input data. Notwithstanding these successes, a fundamental challenge confronting visual RL agents persists — achieving generalization.
To overcome this obstacle, several visual RL generalization benchmarks have emerged, including Progen [6], Distracting Control Suite [47], and DMC-GB [21]. While these benchmarks have been indispensable to visual RL generalization progress, they are not exempt from inherent limitations that pose challenges to further development. Progen offers a diverse distribution of environment configurations and visual appearances. However, it is limited to video games with non-realistic images and low-dimensional discrete action spaces, resulting in a significant gap between its environments and real-world scenarios. Another instance, DMC-GB, is sometimes treated as a golden standard for many state-of-the-art visual generalization algorithms.
Figure 1: The novel RL benchmark for visual generalization. We show that RL-ViGen supports a wide range of tasks with different generalization categories. The algorithms can be evaluated more comprehensively and achieve more convincing experimental results.
Nevertheless, the narrow scope of task classes and generalization categories in existing setups cannot thoroughly and comprehensively evaluate the agent's generalization ability. In addition, although Distracting Control Suite contains two generalization types, it falls short in diversity and complexity. The prevailing trend in this field is to showcase the superiority of proposed algorithms on these benchmarks, which adversely poses a certain risk of promoting overfitting to these benchmarks, rather than discovering algorithms potentially beneficial for solving real-world problems.
In this paper, we introduce a novel Reinforcement Learning benchmark for Visual Generalization (RL-ViGen), presenting numerous merits over existing counterparts. Our benchmark integrates a spectrum of task categories with realistic image inputs, including table-top manipulation, locomotion, autonomous driving, indoor navigation, and dexterous hand manipulation, allowing for a more comprehensive evaluation of the agents' efficacy. Moreover, by incorporating various key aspects in visual RL generalization, such as visual appearances, lighting changes, camera views, scene structures, and cross embodiments, RL-ViGen enables a comprehensive examination of agents' generalization ability against distinct visual conditions.
It is noteworthy that we provide a unified framework that encompasses various state-of-the-art visual RL and generalization algorithms with the same optimization scheme for each approach. The framework not only promotes fair benchmarking comparisons but also lowers the entry barrier for devising novel approaches.
In summary, our contributions are as follows: 1) we propose a novel visual RL generalization benchmark RL-ViGen with diverse, realistic rendering tasks and numerous generalization types; 2) we implement and evaluate various algorithms within a unified framework, enabling a comprehensive analysis of their generalization performance; 3) we conduct comprehensive and extensive experiments to demonstrate the distinct performance of existing approaches when tackling diverse tasks and generalization types, and highlight the benefits and the limitations of current generalizable visual RL algorithms. With all the contributions combined, RL-ViGen may pave the way for further advancements in visual RL generalization, ultimately leading to more robust and adaptable algorithms for real-world applications.
Table 1: Generalization Categories. The following table outlines the types of generalization incorporated within each task. Except for categories considered as not applicable (N/A) (e.g., for locomotion, changes in scene structures are not required), all potential types are included.
Generalization, Categories	Visual
Appearances	Camera
Views	Lighting
Changes	Scene
Structures	Cross
Embodiments
Autonomous Driving	✓	✓	✓	✓	✓
Table-top Manipulation	✓	✓	✓	N/A	✓
Indoor Navigation	✓	✓	✓	✓	N/A
Dexterous Manipulation	✓	✓	✓	N/A	✓
Locomotion	✓	✓	✓	N/A	✓
2 RL-ViGen
RL-ViGen consists of 5 distinct task categories, spanning the domain of locomotion, table-top manipulation, autonomous driving, indoor navigation, and dexterous hand manipulation. In contrast to prior benchmarks, RL-ViGen employs a diverse array of task classes for evaluating the agent's generalization performance. We believe that only through comprehensive examination from multiple perspectives can we obtain convincing results. Furthermore, as shown in Table 1, our benchmark offers a wide range of generalization categories, including visual appearances, camera views, variations in lighting conditions, scene structures, and cross embodiments settings, thereby providing a thorough evaluation of algorithms' robustness and generalization abilities.
2.1 Environments
Dexterous manipulation: Adroit [44] is a sophisticated environment that is explicitly tailored for dexterous hand manipulation tasks. It demands considerable exploration and fine-grained feature capturing due to the sparse reward nature of the environment and the complexity of high-dimensional action space. In RL-ViGen, we have enriched the Adroit environment by integrating diverse visual appearances, camera perspectives, hand types, lighting changes, and object shapes.
Autonomous driving: CARLA [9] serves as a realistic and high-fidelity simulator for autonomous driving, which investigates the control capabilities of agents under dynamic conditions. It has been successfully deployed on visual RL settings in prior studies. Contrary to previous work [24], RL-ViGen provides an enhanced range of dynamic weather and more complex road conditions in different scene structures. Furthermore, flexible camera angle adjustments are also included within RL-ViGen.
Indoor navigation: As an efficient and photorealistic 3D simulator, Habitat [46] combines numerous visual navigation tasks. Succeeding in these tasks requires the agents to own the capability of scene understanding. RL-ViGen builds upon the skokloster-castle scene and proposes additional scenarios with different visual and lighting settings. In addition, the camera view and scene structure are designed to be adjustable.
Table-top manipulation: Robosuite [64] is a modular simulation platform designed to support robot learning. It inherently contains interfaces designed to adjust various scene parameters. Recent work [12] has leveraged this platform to test the agent's generalization ability of visual background changes. RL-ViGen further incorporates dynamic backgrounds, adaptive lighting conditions, and options for embodiment variation, refining the simulation to be closer to the real world.
Locomotion: DeepMind Control is a popular continuous visual RL benchmark. DMC-GB [22] is developed on it and has become a widely used benchmark for evaluating generalization algorithms. Building upon DMC-GB, RL-ViGen introduces objects and corresponding tasks from sophisticated real-world locomotion and manipulation applications, such as the Unitree, Anymal quadrupedal robots, and the Franka Arm. What's more, RL-ViGen also offers a variety of generalization categories to further enrich this environment.
More detailed implementations and modifications can be found in Appendix B and our codebase.
Figure 2: Generalization procedure. The agent is first trained in Stage 1 with a certain fixed scenario. Subsequently, in Stage 2, the agent is tested across various visual generalization scenes in a zero-shot manner. The better the agent performs in various scenes of Stage 2, the stronger generalization ability it demonstrates.
2.2 Generalization Categories
Here, we emphasize the primary generalization categories utilized in RL-ViGen:
Visual appearances: Maintaining effective performance in the presence of altered visual features of objects, scenes, or environments is of vital importance, particularly for visual reinforcement learning. In our benchmark, different components within the environment can be modified with a wide range of colors. Meanwhile, the dynamic video background is also introduced as a challenging setting.
Camera views: In the real world, the agents have to cope with camera configurations, angles, or positions that may not align with those experienced during training. We offer access to set the cameras at different angles, distances, and FOVs. In addition, the number of cameras can be adjusted accordingly.
Lighting conditions: The change in lighting conditions will occur inevitably in the real world. To equip agents with the ability to adapt to such variations, our benchmark supplies interfaces related to the lighting, such as varied light intensity, colors, and dynamical shadow changes.
Scene structures: Mastering the ability of understanding and adapting to different spatial arrangements and organization patterns within various scenes is crucial for a truly generalizable agent. To this end, our benchmark enables modifications in scene structure via adjusting maps, patterns, or introducing extra objects.
Cross embodiments: Adapting learned skills and knowledge to different physical morphologies or embodiments is essential for an agent to perform well across various platforms or robots with different kinematic structures and sensor configurations. Therefore, our benchmark also provides access to modify the embodiment of trained agents in the aspects of model types, sizes, and other physical properties.
3 Algorithmic Baselines for Generalization in Visual RL
3.1 A Unified Framework
Another key contribution of our work is the implementation of a unified codebase to support comparison among various visual RL algorithms. In previous studies, different algorithms adopt distinct optimization schemes, RL baselines, and hyperparameters. For example, SRM [25] and SVEA [23] rely on SAC-based RL algorithms, while PIE-G [61] utilizes a DDPG-based approach. Moreover, mi
nor different implementations could substantially impact the final performance. Therefore, providing a unified framework is of great importance in this domain, enabling more persuasive conclusions to be drawn from evaluating algorithms across a consistent framework and diverse tasks.
3.2 Visual RL Algorithms
In our benchmark, we assemble eight leading visual RL algorithms and apply the same unified training and evaluation framework. DrQ-v2 [57] is the prior state-of-the-art DDPG-based model-free visual RL algorithm in terms of sample efficiency. DrQ [32] is another SAC-based sample efficient visual RL algorithm, which is the base of DrQ-v2. CURL [33] utilizes a SimCLR-style [5] contrastive loss to obtain better visual representations. VRL3 [50] is the state-of-the-art algorithm in Adroit tasks with human demonstrations. The other four algorithms concentrate on achieving robust representations. SVEA [23] employs the Q-value of un-augmented images as the target objective while utilizing data augmentation for reducing the Q-variance; SRM [25] adopts augmentation in the frequency domain to selectively eliminate a part of the observation frequency; PIE-G [61] incorporates ImageNet [8] pre-trained model to further boost the generalization ability; SGQN [3] identifies critical pixels for decision-making via integrating with the saliency map.
In this section, we try to investigate the generalization ability of different approaches in the proposed RL-ViGen benchmark. As shown in Figure 2, all agents are trained in the same fixed training environment and evaluated within various unseen scenarios in a zero-shot manner. The training sample efficiency and asymptotic performance are shown in Appendix F.4. For each task, we evaluate over 5 random seeds and report the mean scores and  $95\%$  confidence intervals. In terms of each trained environment, we present the aggregated scores of the multiple subtasks. The detailed and extensive experimental results can be found in Appendix B and F. The visualization of each environment and generalization types are shown in Appendix D.1.
4.1 Visual Appearances and Lighting Changes
4.1.1 Indoor Navigation
Within the Habitat platform, we choose the ImageNav task and modify the 3D scanned models to introduce novel scenarios with various visual appearances and lighting conditions. We conduct 10 evaluations in each of the 10 selected scenarios (100 trials in total). In contrast to most existing benchmarks, the Habitat-rendered images are captured from a first-person viewpoint by the high-performance 3D simulator. Hence, it can deliver a visualization more akin to real-world scenes. As shown in Figure 3, the superior performance of PIE-G can be attributed to the integration with the ImageNet pre-trained model, equipping PIE-G with a wealth of authentic images and enabling it to handle these scenarios
Figure 3: Generalization score of indoor navigation. We present the success rate of each method. The result indicates that PIE-G achieves better generalization performance on Habitat.
more efficiently. Conversely, consistent with the conclusion drawn from Section 4.1.2, SGQN, which intends to segment the centered agent via eliminating the redundant background, is proved ineffective in these object-rich and first-person view tasks.
4.1.2 Autonomous Driving
Regarding CARLA, we adopt the reward function setting in Zhang et al. [62] and apply a first-person perspective to better resemble real-world driving conditions. As shown in Figure 11 in Appendix D.1, this environment is divided into three levels: Easy, Medium, and Hard. The main modifications involve varying factors such as rainfall intensity, road wetness, and lighting. The higher the disparity from the training scenarios, the more challenging the difficulty level. In this task, one of its distinctive
features is that the input image frequency undergoes considerable changes. Consequently, the SRM approach, which applies data augmentation in the frequency domain, demonstrates the best performance as it can adapt to the input images with varying frequencies. While PIE-G incorporates the ImageNet pre-trained model, its source training images mainly possess higher-frequency features, thus suffering from suboptimal generalization when facing low-frequency scenarios (e.g., dark night). Moreover, SGQN, which extracts salient information, exhibits a decrease in performance when faced with visually rich scenes where the controlled agent is not present in the observed frame. It also should be noted that DrQ gains a degree of generalization ability in this environment. Our observations suggest that since DrQ is a SAC-based algorithm, it tends to be prone to entropy collapse [57]. This implies that the trained agent only produces a single distribution of action in response to diverse image inputs.
Figure 4: Aggregated generalization score of autonomous driving. We present the aggregated return of each method. SRM exhibits better performance to adapt to scenarios where image frequency varies dramatically.
4.1.3 Dexterous Hand Manipulation
In the Adroit environment, we assess the performance of each approach in three single-view tasks: Door, Hammer, and Pen. Since DrQ-v2 and DrQ barely perform well in these challenging environments, we utilize VRL3 [50], the state-of-the-art method in this domain, as the base algorithm and the visual RL approaches in RL-ViGen are re-implemented upon it.
With respect to sample efficiency, it is commonly believed that applying strong augmentation will negatively affect sample efficiency. However, as illustrated in Figure 5 in Appendix F.4, it is worth noting that since VRL3 specifically designs a safe Q mechanism to prevent potential Q divergence for this environment, the generalization algorithms applying strong augmentation can achieve performance comparable to those using only random shift.
Figure 5: Sample efficiency of Adroit. The success rate of each algorithm. We normalize the training steps into (0, 1). The approaches with strong augmentation can also gain comparable performance.
As for generalization, Adroit tasks require agents to identify fine-grained features for dexterous and sophisticated manipulation. Therefore, PIE-G, which leverages ImageNet pre-trained models
Figure 6: The aggregated generalization score of dexterous manipulation. We present the aggregated success rate of each method. PIE-G equipped with the ImageNet pre-trained model exhibits better adaptability to Adroit tasks which necessitate fine-grained information capture.
to capture detailed information, demonstrates the effectiveness of assisting the learned agent in executing downstream tasks, particularly in the hard setting. Moreover, as illustrated in Figure 6, the absence of additional objectives to mitigate the effect of visual changes causes both VRL3 and CURL to struggle in adapting to novel visual situations in these demanding tasks.
4.2 Scene Structures
Generalizable agents that are capable of delivering robust performance across diverse scene structures are essential for potential broad real-world applications. We select CARLA as the testbed for evaluating the generalization of scene structures. The agents are trained in standard training scenarios (highways), and tested in more complex structure settings, including narrow roads, tunnels, and roundabouts with HardRainSunset weather conditions. As shown in Figure 7, the performance of all algorithms falls short of expectations, suggesting that the current visual RL algorithms and generalization approaches are not adequately robust to scene structural changes. More in-depth investigations must be pursued in order to enhance the generalization ability of trained agents to perceive the changing scene structures.
Figure 7: Generalization score of Scene Structure. Across this category of generalization, all algorithms demonstrate unsatisfactory performance.
4.3 Camera Views
We proceed to evaluate the generalization in terms of camera views in the Adroit Environment. As illustrated in Figure 8, under the Easy setting, PIE-G and SGQN exhibit leading generalization capabilities with respect to camera view, while other algorithms also demonstrate some
Figure 8: Generalization score of Camera Views. SGQN indicates advantageous generalization ability across different levels in camera-view generalization.
degree of generalization due to the use of random shift augmentation. However, in the Hard setting, which introduces substantial changes in camera position, orientation, and field of view (FOV), nearly all algorithms, except for SGQN, lose their generalization ability. The exceptional performance of SGQN is mainly due to its heavy reliance on producing saliency maps, which enhances the agent's
self-awareness of object geometry and relative positioning. Hence, this property strengthens its generalization performance even in the face of major camera view alterations.
4.4 Cross Embodiments
Addressing the embodiment mismatch from visual input is crucial, as the embodiment composes a substantial portion of the image and significantly influences robot behavior of interacting with the world. To investigate this type of generalization, Robosuite is employed as the evaluation platform. We utilize the OSC Pose controller [41] during training to facilitate the maintenance of action space dimensions and their respective meanings. Then, the trained agents transfer from Panda Arm to two different morphologies: KUKA IIWA and Kinova3. As illustrated in Figure 9, the overall performance of all algorithms is suboptimal; however, generalization-based
methods, which contain more diverse information during training, exhibit a slight advantage over those primarily focused on sample efficiency in the cross-embodiment setting.
Figure 9: The aggregated generalization score of Cross Embodiments. No algorithm has demonstrated the capability to manage the cross-embodiment generalization yet.
In summary, our experiments reveal that the findings based on previous benchmarks may not accurately reflect the actual progress, leading to a distorted perception of the situation; those advanced visual RL algorithms, previously perceived as cutting-edge, display less efficacy within RL-ViGen. We summarize the main takeaways as follows:
Takeaway 1. The experimental results reveal the varying generalization performance of different visual RL algorithms in distinct tasks and generalization categories, with no single algorithm demonstrating universally strong generalization abilities.
Takeaway 2. Solely enhancing training performance fails to guarantee an improvement in the generalization ability. Although  $\mathrm{DrQ}(\mathrm{v2})$  and  $\mathrm{CURL}$  exhibit high sample efficiency during training and even achieve better asymptotic performance (Appendix F.4), their performance in various generalization scenes has yet to reach a satisfactory level. Therefore, when attempting to improve the generalization ability of an agent, it is crucial to introduce additional inductive biases to aid the training process.
Takeaway 3. An effective generalizable visual RL agent must demonstrate exceptional performance across multiple generalization categories. Previous work has primarily focused on generalization concerning visual appearances, while our experiments reveal considerable shortcomings of existing algorithms in the setting of cross embodiments and scene structures. These underperforming generalization categories go beyond altering the observation space within the Markov Decision Process (MDP); they also bring modifications to the action space and transition probabilities, thus presenting the agent with extra challenges.
Takeaway 4. Each generalization algorithm possesses its own unique strengths. Notably, PIE-G demonstrates superior performance with respect to visual appearances and lighting condition changes, while SRM, under significant image frequency variations, exhibits remarkable robustness. SGQN retains its generalization capacity when facing considerable camera view alterations. In addition, SVEA, without the need for additional parameters and with only minimal modifications, can achieve a certain level of generalization abilities. We hypothesize that stronger performance might be attained through a fusion of different algorithms, such as utilizing pre-trained models with frequency-based augmentation to induce further improvement.