SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is necessary to identify a controllable variable that can approximate the control over the hardness level of DA operations. Inspired by previous research [18], demonstrating the strength of a transformation affects its hardness, we aim to investigate the relationship between the strength of individual DA operations and their corresponding hardness. We perform validation experiments in the Cartpole Balance task, where the visual RL algorithm can achieve satisfactory performance even in the absence of DA. As depicted in Figure 13, we observe a strong linear correlation between the strength and hardness of different DA operations. Therefore, it is reasonable to manipulate the strength of DA operations in order to control their hardness level and thereby investigate the impact of hardness on the effectiveness of DA for sample-efficient visual RL.
B.2 Strength Diversity and Spatial Diversity
Diversity of the strength can be achieved by defining distinct ranges of hardness, as illustrated in Figure 14 (left). During the training process, each image is augmented with a randomly sampled strength value from the specified range. Similarly, spatial diversity is manipulated by defining a set
Figure 14: Illustration of the manipulation of strength diversity and spatial diversity in DA.
of allowable spatial operations, with the size of the set determining the level of spatial diversity. As depicted in Figure 14 (right), a spatial diversity of 1 indicates that only the Translate operation is allowed to move the observation in a single pre-determined direction.
C Detailed Experiment Results of the Rethinking Part
In this section, we present a more detailed description of the experimental setup implemented in Section 3.2 and provide comprehensive results from comparative experiments.
C.1 Customized DA with Controllable Hardness and Diversity
To investigate the impact of DA attributes on the sample efficiency of visual RL, it is necessary to rigorously control variables, ensuring that while one attribute is changed, the other attributes remain constant. For individual DA operations with predetermined type diversity, it is important to accurately control their level of hardness and spatial diversity. Inspired by the experimental design in [18], we propose PadResize-HD, CropShift-HD, and Translate-HD, which enable precise control over the hardness level through strength and the spatial diversity level through degree of freedom.
- CropShift-HD operates by randomly selecting a region in the image and subsequently shifting it to a random position within the input space. The shape of the cropped region can be square or rectangular. The strength of CropShift-HD is determined by the parameter  $H$ , which represents the total number of cropped rows and columns. For instance, when setting the strength to 8, CropShift-HD removes lines from the left, right, top, and bottom borders of the image, where the total number of removed lines is equal to 8. This strength parameter allows for precise control over the hardness level and strength diversity of the operation. Furthermore, during initialization, we pre-sample  $D$  sets of operation parameters for crop and shift to control the spatial diversity of CropShift-HD. Subsequently, during the actual operations, the parameters are randomly selected from the pre-initialized  $D$  sets.
- PadResize-HD starts by adding a specified number of pixels in each direction of the image and thenizes the image back to its original size. Its hardness level is determined by the cumulative number of pixels added in each direction (i.e., top, bottom, left, and right). Given the hardness level, we pre-sample  $D$  sets of pad parameters for each direction to control the spatial diversity of PadResize-HD. Note that when employing Rand PR, as proposed in Section 4, which offers unrestricted spatial diversity, there is no requirement to pre-sample  $D$  sets of pad parameters. Instead, the parameters for each operation are randomly sampled from the entire sets of possibilities each time the operation is executed.
- Translate-HD involves shifting the observations by a certain number of pixels in both the horizontal and vertical directions. The strength of the Translate-HD is defined as the total number of pixels moved in both directions, while its spatial diversity is determined by the allowed directions of movement. Specifically, all the allowed directions include top  $\uparrow$ , bottom  $\downarrow$ , left  $\leftarrow$ , right  $\rightarrow$ , top-left  $\leftarrow$ , top-right  $\nearrow$ , bottom-left  $\swarrow$ , and bottom-right  $\searrow$ , providing eight degrees of freedom. We randomly select  $D$  directions from these options to control the spatial diversity.
C.2 Experimental Setup of the Rethinking Part
We conduct the investigation in the DeepMind Control (DMC) suite [22] and based on the previously superior DrQ-V2 algorithms [5]. We maintain all hyperparameters from DrQ-V2 [5] unchanged and solely manipulate its DA operations. For a detailed network architecture and hyperparameter configuration, please refer to Appendix D.1.
C.3 Additional Results of Hardness Investigation
We present the detailed training curves for different hardness levels of DA operations and the corresponding trend of final performance variation with respect to hardness levels in Figure 15 for the Cup Catch task, Figure 16 for the Finger Spin task, and Figure 17 for the Cheetah Run task.
Cup Catch (CropShift-HD, Strength D=2, Spatial D=1)
Cup Catch (CropShift-HD, Strength D=2, Spatial D=2)
Cup Catch (CropShift-HD, Strength D=2, Spatial D=4)
Cup Catch (CropShift-HD, Strength D=2, Spatial D=8)
Cup Catch (PadResize-HD, Strength  $\mathrm{D} = 3$ , Spatial  $\mathrm{D} = 1$ )
Cup Catch (PadResize-HD, Strength  $\mathrm{D} = 3$ , Spatial  $\mathrm{D} = 2$ )
Cup Catch (PadResize-HD, Strength D=3, Spatial D=4)
Cup Catch (PadResize-HD, Strength D=3, Spatial D=8)
Cup Catch (Translate-HD, Strength D=2, Spatial D=1)
Cup Catch (Translate-HD, Strength D=2, Spatial D=2)
Cup Catch (Translate-HD, Strength D=2, Spatial D=4)
Augmentation Hardness/Strength
Figure 15: Detailed comparison of training sample efficiency and final performance across different hardness levels using three individual DA operations on the Cup Catch task.
Cup Catch (Translate-HD, Strength D=2, Spatial D=8)
Average Type
Figure 16: Detailed comparison of training sample efficiency and final performance across different hardness levels using three individual DA operations on the Finger Spin task.
Cheetah Run (CropShift-HD, Strength  $\mathrm{D} = 2$ , Spatial  $\mathrm{D} = 1$ )
Cheetah Run (CropShift-HD, Strength  $\mathrm{D} = 2$ , Spatial  $\mathrm{D} = 2$ )
Cheetah Run (CropShift-HD, Strength D=2, Spatial D=4)
Cheetah Run (CropShift-HD, Strength D=2, Spatial D=8)
Cheetah Run (PadResize-HD, Strength  $D = 3$ , Spatial  $D = 1$ )
Cheetah Run (PadResize-HD, Strength  $D = 3$ , Spatial  $D = 2$ )
Cheetah Run (PadResize-HD, Strength D=3, Spatial D=4)
Cheetah Run (PadResize-HD, Strength D=3, Spatial D=8)
Cheetah Run (Translate-HD, Strength D=2, Spatial D=1)
Cheetah Run (Translate-HD, Strength D=2, Spatial D=2)
Cheetah Run (Translate-HD, Strength D=2, Spatial D=4)
Figure 17: Detailed comparison of training sample efficiency and final performance across different hardness levels using three individual DA operations on the Cheetah Run task.
Augmentation Hardness/Strength
Cheetah Run (Translate-HD, Strength D=2, Spatial D=8)
Average Type
C.4 Additional Results of Strength Diversity Investigation
We present additional comparative results across varying strength diversity in Figure 18 for the Cup Catch, Finger Spin and Cheetah Run tasks.
Figure 18: Detailed comparison of the sample efficiency across different strength diversity levels.
C.5 Additional Results of Spatial Diversity Investigation
Detailed training curves depicting the performance for various levels of spatial diversity in DA operations, along with the corresponding trend of final performance with increasing spatial diversity, are presented in Figure 19 for the Cup Catch task, Figure 20 for the Finger Spin task, and Figure 21 for the Cheetah Run task.
Cup Catch (CropShift-HD, Hardness=1, Strength D=2)
Cup Catch (PadResize-HD, Hardness=1, Strength D=3)
Cup Catch (Translate-HD, Hardness=1, Strength D=2)
Cup Catch (CropShift-HD, Hardness=2, Strength D=2)
Cup Catch (PadResize-HD, Hardness=2, Strength D=3)
Cup Catch (Translate-HD, Hardness=2, Strength D=2)
Cup Catch (CropShift-HD, Hardness=3, Strength D=2)
Cup Catch (PadResize-HD, Hardness=3, Strength D=3)
Cup Catch (Translate-HD, Hardness=3, Strength D=2)
Cup Catch (CropShift-HD, Hardness=4, Strength D=2)
Cup Catch (PadResize-HD, Hardness=4, Strength D=3)
Cup Catch (Translate-HD, Hardness=4, Strength D=2)
Cup Catch (CropShift-HD, Hardness=5, Strength D=2)
Cup Catch (PadResize-HD, Hardness=5, Strength D=3)
Cup Catch (Translate-HD, Hardness=5, Strength D=2)
Figure 19: Detailed comparison of training sample efficiency and final performance across different spatial diversity levels on the Cup Catch tasks.
Finger Spin (CropShift-HD, Hardness=1, Strength D=2)
Finger Spin (PadResize-HD, Hardness=1, Strength D=3)
Finger Spin (Translate-HD, Hardness=1, Strength D=2)
Finger Spin (CropShift-HD, Hardness=2, Strength D=2)
Finger Spin (PadResize-HD, Hardness=2, Strength D=3)
Finger Spin (Translate-HD, Hardness=2, Strength D=2)
Finger Spin (CropShift-HD, Hardness=3, Strength D=2)
Finger Spin (PadResize-HD, Hardness=3, Strength D=3)
Finger Spin (Translate-HD, Hardness=3, Strength D=2)
Finger Spin (CropShift-HD, Hardness=4, Strength D=2)
Finger Spin (PadResize-HD, Hardness=4, Strength D=3)
Finger Spin (Translate-HD, Hardness=4, Strength D=2)
Finger Spin (CropShift-HD, Hardness=5, Strength D=2)
Finger Spin (PadResize-HD, Hardness=5, Strength D=3)
Finger Spin (Translate-HD, Hardness=5, Strength D=2)
Figure 20: Detailed comparison of training sample efficiency and final performance across different spatial diversity levels on the Finger Spin tasks.
Cheetah Run (CropShift-HD, Hardness=1, Strength D=2)
Cheetah Run (PadResize-HD, Hardness=1, Strength D=3)
Cheetah Run (Translate-HD, Hardness=1, Strength D=2)
Cheetah Run (CropShift-HD, Hardness=2, Strength D=2)
Cheetah Run (PadResize-HD, Hardness=2, Strength D=3)
Cheetah Run (Translate-HD, Hardness=2, Strength D=2)
Cheetah Run (CropShift-HD, Hardness=3, Strength D=2)
Cheetah Run (PadResize-HD, Hardness=3, Strength D=3)
Cheetah Run (Translate-HD, Hardness=3, Strength D=2)
Cheetah Run (CropShift-HD, Hardness=4, Strength D=2)
Cheetah Run (PadResize-HD, Hardness=4, Strength D=3)
Cheetah Run (Translate-HD, Hardness=4, Strength D=2)
Cheetah Run (CropShift-HD, Hardness=5, Strength D=2)
DA Spatial Diversity
Figure 21: Detailed comparison of training sample efficiency and final performance across different spatial diversity levels on the Cheetah Run tasks.
Cheetah Run (PadResize-HD, Hardness=5, Strength D=3)
Average Type
Cheetah Run (Translate-HD, Hardness=5, Strength D=2)
C.6 Additional Results of Type Diversity Investigation
We present additional results in Table C.6 to showcase the detailed performance of employing multi-type DA operations with three different fusion schemes.
Table 3: Detailed results of employing three different fusion methods in the Quadruped Run task, including the mean and standard deviation of performance across 10 random seeds.
Composing-based Fusion Scheme	PC+PR	PC+CS	PC+Tr	PC+Ro	PC+Co	PR+CS	PR+Tr
504.7 ± 141	461.7 ± 105	354.0 ± 221	419.6 ± 221	350.7 ± 158	468.2 ± 165	399.2 ± 240
PR+Ro	PR+Co	CS+Tr	CS+Ro	CS+Co	Tr+Ro	Tr+Co	Ro+Co
409.8 ± 173	369.7 ± 140	395.2 ± 168	463.1 ± 157	331.5 ± 146	382.5 ± 215	302.6 ± 164	263.6 ± 173
Sampling-based Fusion Scheme	PC+PR	PC+CS	PC+Tr	PC+Ro	PC+Co	PR+CS	PR+Tr
525.7 ± 58	490.8 ± 143	487.7 ± 145	492.8 ± 126	351.7 ± 149	475.8 ± 159	497.1 ± 113
PR+Ro	PR+Co	CS+Tr	CS+Ro	CS+Co	Tr+Ro	Tr+Co	Ro+Co
491.6 ± 98	357.3 ± 172	446.6 ± 113	417.5 ± 149	298.0 ± 166	430.1 ± 161	349.0 ± 149	256.7 ± 159
Mixing-based Fusion Scheme	PC+PR	PC+CS	PC+Tr	PC+Ro	PC+Co	PR+CS	PR+Tr
406.1 ± 164	391.8 ± 156	374.2 ± 215	413.4 ± 198	334.4 ± 173	411.1 ± 97	388.4 ± 213
PR+Ro	PR+Co	CS+Tr	CS+Ro	CS+Co	Tr+Ro	Tr+Co	Ro+Co
401.3 ± 177	331.7 ± 165	366.3 ± 173	400.4 ± 143	339.9 ± 171	341.3 ± 131	312.2 ± 161	288.1 ± 112
D Detailed Experimental Setup of the Evaluation Part
In this section, we provide comprehensive implementation details of our experimental setup for evaluating the performance in both DMC and CARLA environments. This includes the introduction of benchmarks, network architecture, and hyperparameters.
D.1 Setup of DeepMind Control Suite
Benchmarks. We conduct evaluations on 12 challenging continuous control tasks, following the task setting of DrQ-V2 [5]. These tasks cover different types of tasks and various challenging elements. A detailed description is presented in Table D.1.
Table 4: Detailed descriptions of the 12 DMC tasks leveraged for evaluation.
Task	Traits	dim(S)	dim(A)
Acrobot Swingup	diff. balance, dense	4	1
Cartpole Swingup Sparse	swing, sparse	4	1
Cheetah Run	run, dense	18	6
Finger Turn Easy	turn, sparse	6	2
Finger Turn Hard	turn, sparse	6	2
Hopper Hop	move, dense	14	4
Quadruped Run	run, dense	56	12
Quadruped Walk	walk, dense	56	12
Reach Duplo	manipulation, sparse	55	9
Reacher Easy	reach, dense	4	2
Reacher Hard	reach, dense	4	2
Walker Run	run, dense	18	6
Hyper-parameters. To demonstrate the general applicability of our method, we keep all environment-specific hyperparameters from DrQ-V2 [5] unchanged and solely replace the original DA operations with Rand PR and CycAug. The hyper-parameters of DA operations are also
presented in Table 5. While we aim to maintain consistent settings across all tasks, we observe that the cycling interval of CycAug needed to be adjusted for more challenging tasks. Consequently, we set the cycling interval to  $2 \times 10^{5}$  steps for the Quadruped Walk, Quadruped Run and Hopper Hop tasks. CycAug, a naive attempt guided by the insights that effective multi-type DA fusion in sample-efficient visual RL should consider the data-sensitive nature of RL, demonstrates the effectiveness of reducing the frequency of DA operation switching for training stability. This aligns with our rethinking conclusions. Our attempt provides preliminary validation of the feasibility of employing multi-type DA operations in sample-efficient visual RL, surpassing the limitations of individual DA approaches. There is still significant potential for future research to explore and refine more suitable fusion schemes.
Table 5: A default set of hyper-parameters used in DMC evaluation.
Algorithms Hyper-parameters
Replay buffer capacity	10^6
Action repeat	2
Seed frames	4000
Exploration steps	2000
n-step returns	3
Mini-batch size	256
Discount γ	0.99
Optimizer	Adam
Learning rate	10^-4
Agent update frequency	2
Critic Q-function soft-update rate τ	0.01
Features dim.	50
Repr. dim.	32 × 35 × 35
Hidden dim.	1024
Exploration stddev. clip	0.3
Exploration stddev. schedule	linear(1.0,0.1,500000)
DA Operations Hyper-parameters
Strength range of Rand PR	[0,16]
Cycling interval of CycAug	1 × 10^5 steps
D.2 Setup of CARLA Simulator
**Benchmarks.** Our experimental setup in CARLA is adapted from the work of [66]. Specifically, we employ three cameras mounted on the roof of the vehicle, with each camera capturing a 60-degree field of view, as illustrated in Figure 22. These cameras are utilized for both training the agent and evaluating the final performance of the different methods. The reward function is defined as:
$$
r _ {t} = \mathbf {v} _ {\text {e g o}} ^ {\top} \hat {\mathbf {u}} _ {\text {h i g h w a y}} \cdot \Delta t - \lambda_ {i} \cdot \text {i m p l u s e} - \lambda_ {s} \cdot | \text {s t e e r} |, \tag {2}
$$
where  $\mathbf{v}_{\mathrm{ego}}$  represents the velocity vector of the ego vehicle projected onto the unit vector  $\hat{\mathbf{u}}_{\mathrm{highway}}$  that aligns with the highway direction. It is multiplied by the time discretization  $\Delta t = 0.05$  to measure
Figure 22: Pipeline of DrQ-V2, Rand PR and CycAug in CARLA. The size of the original observations is  $84 \times 252$ , and we first resize them to  $3 \times 84 \times 84$  before applying DA operations. After DA operations, we resize the observations back to their original size and then input them to the network.
the progress of the vehicle on the highway in meters. Collisions are quantified in terms of impulses, measured in Newton-seconds. Additionally, a steering penalty is imposed, with steer  $\in [-1, 1]$ . The reward function incorporates weights  $\lambda_{i} = 10^{-4}$  and  $\lambda_{s} = 1$ .
To adapt the DrQ-V2 algorithms to CARLA tasks, we partition the original observations, which are of size  $84 \times 252$ , into three square images of size  $84 \times 84$ , each corresponding to the content captured by a different camera. These individual images are then subjected to DA operations, as shown in Figure 22. We evaluate our proposed Rand PR and CycAug methods against the PadCrop operation originally used in DrQ-V2 in four different weather settings. These varied weather conditions not only offer a diverse range of realistic observations but also introduce different dynamics within the environments. The Figure 23 illustrates the diverse and realistic observations obtained from the four weather settings we leveraged in our evaluation.
WetNoon
Figure 23: Illustration of realistic observations captured in CARLA under various weather conditions.
SoftRainNoon
HardRainSunset
Hyper-parameters. We adopt various hyperparameter settings from DrQ-V2 in the DMC tasks, only modifying the dimensions of the intermediate layers in the network to accommodate the observation size of CARLA. The complete hyperparameter settings are presented in Table 6.
Table 6: A default set of hyper-parameters used in CARLA evaluation.
Algorithms Hyper-parameters
Replay buffer capacity	10^5
Action repeat	4
Exploration steps	100
n-step returns	3
Mini-batch size	512
Discount γ	0.99
Optimizer	Adam
Learning rate	10^-4
Agent update frequency	2
Critic Q-function soft-update rate τ	0.01
Features dim.	50
Repr. dim.	32 × 35 × 119
Hidden dim.	1024
Exploration stddev. clip	0.3
Exploration stddev. schedule	linear(1.0, 0.1, 100000)
DA Operations Hyper-parameters
Strength range of Rand PR	[0, 16]
Cycling interval of CycAug	2 × 10^4 steps