SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Combined with the takeaways, we hope that an algorithm's success in RL-ViGen can indicate its potential applicability in more complex and unpredictable real-world scenarios. In the future, a
holistic and multi-dimensional approach, encompassing aspects such as scene structures, camera views, and cross embodiments, is critical for fostering truly generalizable agents capable of navigating in varied and dynamic real-world environments. Equally, the design of more sophisticated and realistic training environments which enable to reflect the complexity of real-world conditions, can also serve as a crucial area for future explorations.
RL benchmarks. There exists a multitude of mature benchmarks aiming for evaluating reinforcement learning algorithms [4, 51, 38, 10, 17, 40, 55]. For instance, Atari [2] and Gym-MuJoCo [48] are exemplary benchmarks in deep reinforcement learning. In other subdomains, D4RL [15] serves as a popular benchmark for offline RL algorithms, while URLB [34] provides an evaluating platform with respect to unsupervised RL algorithms. MetaWorld [59] is often used to evaluate multi-task and meta-learning scenarios. SafetyGym [45], meanwhile, is predominantly applied for testing Safe RL algorithms. Recently, MineDojo [13] benchmarks embodied agents in exploration and multi-task domains. Contrasting to these benchmarks, RL-ViGen distinguishes itself by incorporating a variety of task classes and an array of generalization categories and primarily focuses on evaluating agents' visual generalization abilities.
Generalization. How to endow models' generalization abilities is a pivotal topic in machine learning. In computer vision, well-established benchmarks are available for exploring generalization problems [54, 53, 58, 31]. While several approaches have been proposed in RL and robotics to tackle such issues [23, 61, 25, 3, 63, 29, 30, 28, 56, 60], the benchmarks in use are relatively immature [11, 6, 47, 22], fraught with numerous limitations, and lacking a unified framework for comparison. For example, Progen [6] is a widely used benchmark for quantifying the agents' generalization abilities. However, Progen remains a video game platform, with the human-imaged world rather real-world counterparts, offering limited assistance for agents' generalization in real-world scenarios. The Distracting Control Suite [47] and DMC-GB [21], building upon DM-Control [49], introduce some types of visual distractions. Nevertheless, their tasks are solely focused on locomotion, and there remains a substantial gap between this simulated environment and real-world scenarios. By contrast, RL-ViGen encompasses various forms of generalization, exhibits a high degree of photorealism, and includes a diverse range of tasks. Avalon [1] is another valuable benchmark for RL generalization. It shares a unified world dynamics and task structure, making it highly suitable as a benchmark for in-distribution generalization. Contrary to Avalon, which is mainly concerned with task-level generalization, RL-ViGen mainly focuses on out-of-distribution generalization, with a specific concentration on the visual aspects of generalization.
7 Conclusion, Limitations, and Future Work
In this work, we propose a novel Reinforcement Learning benchmark for Visual Generalization (RL-ViGen), a comprehensive benchmark for evaluating the visual generalization abilities of trained agents. RL-ViGen stands apart from existing benchmarks by boasting a broader diversity of tasks and generalization categories, which in turn fosters more persuasive conclusions. According to the quantitative experimental results from RL-ViGen, we note that, as of now, there are no existing generalization algorithms that can adeptly manage all tasks and generalization types. It is our expectation that the advent of RL-ViGen will bring fresh perspectives to the research community, and stimulate the advancement of agents that can truly exhibit overall visual generalization capabilities.
Limitations. The agents trained through RL-ViGen have not yet been evaluated in real-world scenarios. In our future work, we would like to build certain real-world tasks to demonstrate the value that RL-ViGen can provide in developing generalizable agents for real-world applications.
This work is supported by research program 2022ZD0161700.
[63] Tony Z. Zhao, Siddharth Karamcheti, Kollar Thomas, Chelsea Finn, and Liang Percy. What makes representation learning from videos hard for control? In https://tonyzhaozh.github.io/2023.
[64] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020.
A Visual Reinforcement Learning Baselines
DrQ: This model-free, off-policy reinforcement learning algorithm, is based on Soft Actor-Critic (SAC) [19]. DrQ enhances training stability via applying data augmentation to regularize the Q value of state-action pairs. The key of DrQ is to promote similarity between augmented state-action pairs. The Q-regularization technique is shown in Eq 1, where  $K$  is the number of samples,  $\mathcal{T}$  is the collection of augmentation.
$$
\mathbb {E} _ {\substack {s \sim \mu (\cdot) \\ a \sim \pi (\cdot | s)}} [ Q (s, a) ] \approx \frac {1}{K} \sum_ {k = 1} ^ {K} Q \left(f \left(s ^ {*}, \nu_ {k}\right), a _ {k}\right) \text {where} \nu_ {k} \in \mathcal {T} \text {and} a _ {k} \sim \pi (\cdot | f (s ^ {*}, \nu_ {k})) \tag{1}
$$
DrQ-v2: An improved version of DrQ. DrQ-v2 fuses essential elements from the DDPG algorithm with data augmentation to strengthen visual RL agents' performance. DrQ-v2 also incorporates techniques such as n-step return and target critic, leading to commendable results in most of the medium and hard level DM-Control tasks. The TD-target is defined as follows, where  $x_{t + n}$  is the n-step observation,  $a_{t + n}$  is the n-step action, and  $\bar{\theta}_{1,2}$  is the Q-target networks:
$$
y = \sum_ {i = 0} ^ {n - 1} \gamma^ {i} r _ {t + i} + \gamma^ {n} \min  _ {k = 1, 2} Q _ {\bar {\theta} _ {k}} (a u g (x _ {t + n}), \boldsymbol {a} _ {t + n}) \tag {2}
$$
CURL: CURL integrates contrastive learning methods into the reinforcement learning training process. The auxiliary contrastive loss (Eq 3) allows the agent to obtain better image representation during training, thus mitigating the optimization difficulty under high-dimensional inputs. In our implementation, we only apply a single encoder to produce visual representations instead of two polyak-averaging encoders. This alteration improves the sample efficiency of CURL and put it on a comparable performance with DrQ-v2. More experiments are shown in Appendix F.3.
$$
\mathcal {L} _ {q} = \log \frac {\exp (q ^ {T} W k _ {+})}{\exp (q ^ {T} W k _ {+}) + \sum_ {i = 0} ^ {K - 1} \exp (q ^ {T} W k _ {i})} \tag {3}
$$
PIE-G: PIE-G proposes a simple yet effective method, combining ImagNet pre-trained visual representations with the early layer and updates of BatchNorm statistical parameters to further enhance the generalization ability of the agent.
SVEA: SVEA finds that heavy data augmentation introduces additional high variance to agent training, which can lead to instability or even divergence. SVEA suggests that using the Q-values of non-augmented images as the target of estimated Q-values for augmented images (Eq 4), thus stabilizing the variance of the value estimation.
$$
\left\| Q _ {\theta} \left(a u g (x _ {t}), \mathbf {a} _ {t}\right) - q _ {t} ^ {\mathrm {t g t}} \right\| _ {2} ^ {2} \tag {4}
$$
SRM: SRM proposes a novel data augmentation method that operates in the frequency domain. It helps diversify data and alleviate distribution shift issues under various visual scenarios. During the training stage, SRM randomly discards parts of the frequency information from observations, forcing the policy to select suitable actions based on the remaining information. The augmentation method is shown in Eq 5, where  $\mathcal{F}$  is the fast Fourier transform,  $\mathbf{M}$  is a binary masking matrix, and  $\mathbf{Z}$  is a random noise image.
$$
\hat {\mathcal {F}} \left(o _ {i}\right) = \mathbf {M} \cdot \mathcal {F} \left(o _ {i}\right) + \left(\mathbf {1} - \mathbf {M}\right) \cdot \mathcal {F} (\mathbf {Z}) \tag {5}
$$
SGQN: This algorithm introduces the saliency map for the use of augmenting images. Saliency maps, a tool used in computer vision, offers an interpretability analysis of encoders. SGQN retains only agent's focusing areas and removes the visual background by the generated saliency map.
This approach utilizes the augmentation objectives in SVEA [23] to further improve the model's generalization performance. The auxiliary objective is shown in Eq 6, where  $M_{\rho}((o,a),a)$  is the binary masking matrix introduced from the saliency map.
$$
L _ {C} (\theta) = \left\| Q _ {\theta} (o, a) - Q _ {\theta} (o \odot M _ {\rho} (o, a), a) \right\| ^ {2} \tag {6}
$$
Indoor navigation: Habitat serves as the simulator and extends a variety of indoor navigation tasks. We select ImageNav as the test env, whose goal is defined by the image of target location in the chosen map. Due to the complexity in the default training and validation episode settings, which demands extensive training periods to achieve a satisfactory standard, we simplify the setup to 500 initial positions and 1 target position. Meanwhile, we utilize the 3D scenes from the Gibson dataset as our map for all experiments.
Autonomous driving: We choose the stable version of CARLA 0.9.10 for simulation. The reward function is adopted from Zhang et al. [62]. We also implement the wrapping methods from Huang et al. [24] for novel CARLA environments. Moreover, to enhance exploration and ensure stable training, we standardized the std_schedule across all algorithms. Each difficulty level contains two weathers, Easy level: soft_high_light, soft_noisy_low_light; Medium level: HardRainSunset, SoftRainSunset; Hard level: hard_low_light, hard_noisy_low_light. The aggregated return is calculated by averaging over the weather at the same level. Further details can be accessed in the documentation provided within our GitHub repository.
Dexterous manipulation: In RL-ViGen, we select three single-view Adroit tasks. Given that tasks in Adroit typically necessitate demonstrations for successful completion, we employ VRL3, the state-of-the-art baseline for these tasks. Since the update process of VRL3 is based on DrQ-v2, it allows a seamless transfer of our algorithms to VRL3's codebase. There are three stages for VRL3 training: stage1 responses to gain a basic perception ability via pretraining on ImageNet; stage2 utilizes offline RL training with expert demonstrations; stage3 executes online training. It is noteworthy to mention that the experiment of VRL3 demonstrates that in single view tasks, only applying stage3 is sufficient to accomplish Adroit tasks with high sample efficiency. Therefore, to compare each algorithm more effectively, we exclude the use of stage1 and stage2. The aggregated success rate is calculated by averaging over all three tasks.
Table-top manipulation: SECANT [12] previously employed Robosuite for generalization testing. Building on its codebase, we adopt one of the latest versions - Robosuite 1.4.0 and mujoco 2.3.0 as well as simplified the installation process. Meanwhile, we also introduce a range of new classes of visual generalization. For each difficulty level, we deploy a variety of scenarios, and each trained agent is evaluated within each environment 10 times (in a total of 100 evaluations). The aggregated return is calculated by averaging over all three tasks.
Locomotion: In addition to the locomotion tasks from DM-Control (1.0.8 version), we also incorporate models from Mujocoreie [7], and carefully designed corresponding rewards, enabling them to accomplish walk or stand tasks. Furthermore, building on DMC-GB, we have added additional generalization categories for further enriching RL-ViGen. The aggregated return is calculated by averaging over two tasks.
Our experiments are all conducted with TeslaA40 or TeslaA100 GPU and AMD EPYC 7542 32-Core Processor CPU. More details can be founded in https://github.com/gemcollector/RL-ViGen.
C Hyper-parameters
We use the same hyper-parameters as the original papers and perform a small-scale grid search to achieve better performance of certain algorithms. The common hyper-parameters are listed in Table 2.
The individual hyper-parameters are listed in the following Tables. The additional hyper-parameters introduced by SGQN are listed as well.
Table 2: Common hyper-parameters in RL-ViGen.
Hyper-parameters	Value
Input size	84 × 84
Discount factor γ	0.99
Replay Buffer size	int(1e7)
Feature dim	DrQ(v2), CURL: 50, otherwise: 256
Action repeat	Robosuite: 1, otherwise: 2
N-step return	DrQ: 1, otherwise: 3
Optimizer	Adam
Hidden dim	1024
Frame stack	3
Table 3: CARLA hyper-parameters in RL-ViGen.
Hyper-parameters	Value
Training Frames	int(1e6)
Learning Rate	PIE-G: 5e-5, DrQ: 5e-4, otherwise: 1e-4
N-step return	1
SGQN quantile	0.9
SGQN critic weight	0.5
SGQN aux lr	8e-5
D Visualization of each difficulty level
To gain a better understanding of our setting and RL-ViGen, we visualize the images under various generalization settings and difficulty levels as mentioned in the experiment section.
D.1 Visual Appearances and Lighting Changes
D.1.1 Robosuite
In the context of Robosuite, each difficulty level comprises 10 distinct scenes. We perform 10 trials for each of these scenes (100 trials in total).
The Easy level includes changes of the background appearance, while the Hard level contains additional complexities of moving light and alterations to the robotic arm's color. The Extreme level further employs a dynamic video background to evaluate the trained agents' generalization abilities. The visualized figures are shown in Figure 10.
Easy
Figure 10: The visualization of various difficulty levels of Robosuite. This figure shows examples from the Door task. As the difficulty level increases, more types of distracting factors are introduced.
D.1.2 CARLA
Apart from the default weather settings in CARLA, we implement a series of challenging new scenarios. In our CARLA setup, each level of difficulty is characterized by two specific weather conditions. The Easy level includes soft_noisy_low_light and soft_high_light, while the Medium
Table 4: Habitat hyper-parameters in RL-ViGen.
Hyper-parameters	Value
Training Frames	int(7e5)
Learning Rate	1e-4
N-step return	1
SGQN quantile	0.93
SGQN critic weight	0.9
SGQN aux lr	8e-5
Table 5: Adroit hyper-parameters in RL-ViGen.
Hyper-parameters	Task	Value
Training Frames	Hammer	int(1e6)
Door	int(1e6)
Pen	int(2e6)
Learning Rate	Hammer	1e-4
Door	1e-4
Pen	1e-4
SGQN quantile	Hammer	0.9
Door	0.9
Pen	0.9
SGQN critic weight	Hammer	0.9
Door	0.5
Pen	0.9
SGQN aux lr	Hammer	8e-5
Door	8e-5
Pen	8e-5
level is defined by the HardRainSunset and SoftRainSunset conditions. The Hard level contains hard_low_light and hard_noisy_low_light. As the disparity between the novel scenarios and the training images increases, the difficulty level of generalization grows. RL-ViGen also encompasses challenging conditions such as rainy, overcast, and slippery road surfaces. The visualized figures are shown in Figure 11.
Medium
Figure 11: The visualization of various difficult level of CARLA. The higher the disparity from the training observations, the more challenging the new scenario.
D.1.3 Habitat
For Habitat, the Gestaltor 3D model editor is applied to modify the appearance of the scene's 3D models. A total of 10 distinct scenarios are created. The visualized figures are shown in Figure 12.
Table 6: Robosuite hyper-parameters in RL-ViGen.
Hyper-parameters	Task	Value
Training Frames	Door	int(6e5)
Lift	int(8e5)
TwoArmPegInhole	int(8e5)
Learning Rate	Door	1e-4
Lift	DrQ(v2), CURL: 1e-4, otherwise: 8e-5
TwoArmPegInhole	SGQN: 1e-4, otherwise: 8e-5
Level	Door	Easy
Lift	Medium
TwoArmPegInhole	Medium
SGQN quantile	Door	0.9
Lift	0.9
TwoArmPegInhole	0.87
SGQN critic weight	Door	0.7
Lift	0.7
TwoArmPegInhole	0.7
SGQN aux lr	Door	8e-5
Lift	8e-5
TwoArmPegInhole	8e-5
Figure 12: The visualization of Habitat. We create 10 distinct scenarios for the generalization of visual appearances.
D.1.4 Locomotion
For DM-Control, we further augment numerous new types of generalizations on the basis of DMC-GB. For the unitree series tasks, Easy and Hard denote two levels of difficulty regarding light color, light position, changes of light's movement and objects' color. The visualized figures are shown in Figure 13.
D.1.5 Adroit
In the Adroit environment, we provide four generalization scenarios. The Color setting changes the background, object color, and table texture, while the Video setting utilizes a dynamic background and introduces moving light. As illustrated in Figure 14, each scenario is configured with two levels of difficulty.
D.2 Camera Views
For camera view generalization, we implement alternations to the camera view through the modification of camera's orientation, position, and FOV. The visualized figure is shown in Figure 15.
Figure 13: The visualization of various difficulty level of DM-Control. The figure above show examples from unitree tasks. Factors such as light color, light position, movement of light, and object color are varied.
Train
Figure 14: The visualization of various difficulty levels of Adroit. This figure demonstrates examples from the Pen task. We show four generalization scenarios provided in RL-ViGen.
Color-Easy
Color-Hard
Video-Easy
Video-Hard
D.3 Scene Structures
As shown in Figure 16, we established a variety of road scenarios in CARLA, including roundabouts, narrow paths, tunnels, etc., which can be also utilized in conjunction with other adjustable parameters. As the experiment illustrated in Figure 7, we employ the same weather conditions as those during training.
D.4 Cross Embodiments
In terms of cross-embodiment generalization, we modify the type of the robotic arm in Robosuite. In addition, by leveraging the OSC_POSE control method, the input actions are interpreted as delta values from the current state, thus facilitating to maintain the action space dimensions and corresponding meanings.
E Visualization of each difficulty level
At first, we will structure the general modifications made to each generalization type. A more detailed configuration of each environment will be specified subsequently.
- Visual appearances: The generalization type of visual appearances are divided into two variations: static and dynamic changes.
- Color: Drawing on DMC-GB [22], we categorize color variations into two levels (easy and hard) of difficulty, each containing 100 color combinations. In both levels, alterations are made to three environmental attributes: body, grid, and skybox. In the easy level, the contrast between the color combinations and the original training scene's color combinations is 0.07, while in the hard level, the contrast increases to 0.14.
- Video: We divide video variations into two levels of difficulty by replacing the environment's skybox with videos. The easy level contains 10 videos, while the hard level
Figure 15: The visualization of camera views of Adroit. The larger the deviation angle of the camera, the higher the difficulty of generalization.
Roundabout
Figure 16: The visualization of scene structures of CARLA. We selected certain locations within different maps to serve as scenarios for scene structure generalization.
consists of 100 videos with increased contrast. Additionally, the hard level also removes the reference plane of the ground.
- Lighting conditions: The generalization type of lighting conditions are divided into two variations: static and dynamic changes. Each variations are categorized into two difficulty levels.
- Static: We adjust the static lighting conditions by altering the position, intensity, and color.
* Position: Easy: the coordinates  $x, y$  will be determined by adding values randomly sampled from the interval [-0.5, 0.5] via uniform sampling to the original light source coordinates, and the  $z$  coordinate will be selected from the range [-0.2, 0.2]; Hard: the coordinates  $x, y$  will be determined by adding values randomly sampled from the interval [-1, 1] via uniform sampling to the original light source coordinates, and the  $z$  coordinate will be selected from the range [-0.5, 0.5].
* Intensity: Easy: the value of intensity will be uniformly sampled from the interval [0.7, 1.4]; Easy: the value of intensity will be uniformly sampled from the interval [1.4, 2.5].
* Color: Easy: the RGB values will be determined by adding values randomly sampled from the interval  $[-0.1, 0.1]$  via uniform sampling to each channel; Hard: the RGB values will be determined by adding values randomly sampled from the interval  $[-0.2, 0.2]$  via uniform sampling to each channel. It should be noted that the RGB values are normalized to [0, 1].
- Dynamic: The alteration of y coordinate is defined by adding a value within the interval [-1.2, 1.2] to the original coordinate value, with an incremental change of 0.04 at each step.
- Camera views: The generalization type of camera views are divided into two variations: static and dynamic changes. Each variations are categorized into two difficulty levels.
- Position: Easy: the coordinates  $x, y, z$  will be determined by adding values randomly sampled from the interval [-0.03, 0.03] via uniform sampling to the original camera positions; Hard: the coordinates  $x, y, z$  will be determined by adding values randomly
IIWA
Figure 17: The visualization of cross embodiment of Robosuite. This figure shows examples from the Door task. Here, we demonstrate our modification of the style of the robotic arm for cross-embodiment generalization.
Kinova3
sampled from the interval [-0.07, 0.07] via uniform sampling to the original camera positions.
- Orientation: Easy: each component of quaternion will be determined by adding values randomly sampled from the interval  $[-0.03, 0.03]$  via uniform sampling to the original camera quaternion; Hard: each component of quaternion will be determined by adding values randomly sampled from the interval  $[-0.08, 0.08]$  via uniform sampling to the original camera quaternion.
- FOV: Easy: the coordinates  $x, y, z$  will be determined by adding values randomly sampled from the interval [-5, 5] via uniform sampling to the original FOV; Hard: the coordinates  $x, y, z$  will be determined by adding values randomly sampled from the interval [-10, 10] via uniform sampling to the original FOV.
Due to the challenges in defining the difficulties associated with the generalization types of the scene structures and cross embodiments, and the complexity in numerically describing the differences between scenes, we don't provide additional quantification to these two types.
F.1 Generalization Evaluation
F.1.1 Locomotion
Built upon DM-Control, which has included numerous locomotion tasks, we extend this benchmark by integrating real-world robot models from Mujocoreie [7] with corresponding tasks. Moreover, RL-ViGen also augments DMC-GB with more tasks and generalization types. Here we evaluate the performance of each algorithm on the Unitree series tasks. Figure 18 demonstrates that all generalization algorithms exhibit comparable performance. More specifically, SVEA outperforms other techniques in the Easy setting, where the other generalization techniques do not show any advantages. In the Hard setting, where the agent's color closely resembles that of the surrounding environment, SGQN may not effectively capture the agent's outline, leading to a performance decline.
F.1.2 Table-top Manipulation