SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guozheng Ma $^{1}$  Linrui Zhang $^{1}$  Haoyu Wang $^{1}$  Lu Li $^{1}$  Zilin Wang $^{1}$  Zhen Wang $^{2}$  Li Shen $^{3*}$  Xueqian Wang $^{1*}$  Dacheng Tao $^{2}$
$^{1}$ Tsinghua University  $^{2}$ The University of Sydney  $^{3}$ JD Explore Academy
{mgz21,haoyuwa22,lilu21,wangzl21}@mails.tsinghua.edu.cn
zwan4121@uni.sydney.edu.au; wang.xq@sz.tsinghua.edu.cn
{zhanglr.auto,mathshenli,dacheng.tao}@gmail.com
Data augmentation (DA) is a crucial technique for enhancing the sample efficiency of visual reinforcement learning (RL) algorithms. Notably, employing simple observation transformations alone can yield outstanding performance without extra auxiliary representation tasks or pre-trained encoders. However, it remains unclear which attributes of DA account for its effectiveness in achieving sample-efficient visual RL. To investigate this issue and further explore the potential of DA, this work conducts comprehensive experiments to assess the impact of DA's attributes on its efficacy and provides the following insights and improvements: (1) For individual DA operations, we reveal that both ample spatial diversity and slight hardness are indispensable. Building on this finding, we introduce Random PadResize (Rand PR), a new DA operation that offers abundant spatial diversity with minimal hardness. (2) For multi-type DA fusion schemes, the increased DA hardness and unstable data distribution result in the current fusion schemes being unable to achieve higher sample efficiency than their corresponding individual operations. Taking the non-stationary nature of RL into account, we propose a RL-tailored multi-type DA fusion scheme called Cycling Augmentation (CycAug), which performs periodic cycles of different DA operations to increase type diversity while maintaining data distribution consistency. Extensive evaluations on the DeepMind Control suite and CARLA driving simulator demonstrate that our methods achieve superior sample efficiency compared with the prior state-of-the-art methods.
Visual reinforcement learning (RL) has shown great potential in various domains, enabling decision-making directly from high-dimensional visual inputs [1]. However, the dual requirements of simultaneously learning compact state representations and optimizing task-specific policies lead to prohibitive sample complexity and massive environment interactions, hindering its practical deployment [2]. Enhancing sample efficiency is a critical and inevitable challenge faced by the entire visual RL community, and data augmentation (DA) has emerged as a remarkably effective approach to tackle this issue [3-7]. As illustrated in Figure 1, even simple random shift transformations applied to input observations can lead to significant performance improvements in previously unsuccessful algorithm [5]. Moreover, it has been proven that DA alone can lead to more
Figure 1: Benefit of DA in visual RL.
*Corresponding authors: Li Shen and Xueqian Wang
efficient performance compared with meticulously crafting self-supervised learning tasks [8] or pre-training representation encoders with extra data [9]. The substantial effectiveness of DA in improving sample efficiency has established it as an essential and indispensable component in the vast majority of visual RL methods [4, 5, 7, 10-12].
However, recent research has predominantly concentrated on integrating DA with other complementary techniques to enhance performance [7, 13-17], while ignoring the opportunity to fully explore and harness the intrinsic potential of DA operations themselves. There is a notable absence of actionable guidelines for designing more effective DA operations specifically tailored to visual RL scenarios. Motivated by this imperative requirement, it is crucial to undertake a thorough investigation and in-depth analysis to explore a fundamental question:
Which attributes enable effective DA for achieving sample-efficient visual RL?
Hence, this study begins with comprehensive experiments in order to assess the essential attributes of DA for enhancing sample efficiency in visual RL. Specifically, we unpack the attributes of DA operations in terms of hardness and diversity, which are two main dimensions that previous studies leveraged to deconstruct DA in other domains [18-21]. The extensive experiments provide strong support for the following key findings, which highlight the distinctive requirements of visual RL: - The training performance of visual RL is highly sensitive to the increase of DA's hardness. - Sufficient spatial diversity is an indispensable attribute for achieving effective DA. - Unlimited increases in strength diversity can harm visual RL performance. - Despite the increased type diversity, naively applying multi-type DA to visual RL training can lead to decreased performance.
Drawing upon these findings, we propose two actionable guidelines and corresponding strategies for enhancing the effectiveness of DA in visual RL. (1) For individual DA operations, we reveal that effective operations should exhibit sufficient spatial diversity with minimal hardness. Building on this insight, we introduce Random PadResize (Rand PR), a new DA transformation that satisfies the essential attributes required for visual RL. Rand PR alleviates the hardness of DA by preventing the loss of information in observations, which is inevitable in the conventional PadCrop transformation that widely used in previous methods such as DrQ-v2 [5]. Additionally, it provides significant spatial diversity by randomly scaling and positioning the original content within the final input space. (2) For multi-type DA fusion schemes, it is imperative to take the data-sensitive nature of RL training into account when designing appropriate fusion schemes in visual RL. Inspired by this guideline, we propose a RL-tailored multi-type DA fusion scheme called Cycling Augmentation (CycAug), which leverages the diversity benefits of multiple DAs by cyclically applying different transformations while mitigating the negative impact on training stability caused by the excessively frequent changes in data distribution. With Rand PR as a key component, CycAug achieves superior sample efficiency on various tasks of the DeepMind Control suite [22]. Moreover, in the challenging low data regime of CARLA [23], CycAug outperforms the previous SOTA algorithm by a substantial margin of  $43.7\%$ .
In the end, our main contributions can be summarized as following three-fold:
1. We conduct extensive experiments to investigate the essential DA attributes for achieving sample-efficient visual RL and highlight the unique requirements of visual RL for DA.
2. We present two actionable guidelines to further exploit the potential of DA, specifically focusing on individual DA operations and multi-type DA fusion schemes. Building upon these guidelines, we propose Rand PR and CycAug as corresponding improvement strategies.
3. CycAug, incorporating Rand PR as a key component, achieves state-of-the-art sample efficiency in extensive benchmark tasks on DM Control and CARLA.
In this section, we provide a succinct overview of prior research conducted in sample-efficient visual RL, as well as DA methods utilized in the broader domain of deep learning.
2.1 Sample-Efficient Visual RL
Visual RL entails learning compact state representations from high-dimensional observations and optimizing task-specific policies at the same time, resulting in a prohibitive sample complexity [2]. To enhance sample efficiency, the first widely adopted approach is leveraging self-supervised learning to promote agent's representation learning. Specifically, various auxiliary tasks have been designed
in previous studies, including pixel or latent reconstruction [2, 14], future prediction [24, 7, 13] and contrastive learning for instance discrimination [16, 17, 15] or temporal discrimination [25-28]. Another approach aimed at facilitating representation learning is to pre-train a visual encoder that enables efficient adaptation to downstream tasks [29-34]. Additionally, a number of model-based methods explicitly construct a world model of the RL environment in either pixel or latent spaces to enable planning [35-41]. Orthogonal to these methods, DA has demonstrated astonishing effectiveness in improving the sample efficiency of visual RL algorithms [10, 4, 5]. Moreover, recent research has focused on effectively integrating DA with other representation learning methods to further enhance sample efficiency [7, 13, 16, 25, 14]. However, the intrinsic potential of DA operations has been inadequately explored. To address this research gap, this study conducts thorough experiments to investigate the fundamental attributes that contribute to the effectiveness of DA and subsequently provide actionable guidelines for DA design. Furthermore, we illustrate that developing more effective DA methods alone can result in a substantial improvement in sample efficiency.
2.2 Data Augmentation (DA)
Over the past decades, DA has achieved widespread recognition and adoption within the deep learning community [42-48]. For clarity, we classify previous studies on DA into two distinct aspects: individual DA operations, which utilize single-type data transformations, and multi-type DA fusion methods, which combine diverse DA operations through specific fusion schemes.
Individual DA Operations. Numerous studies have shown that the adoption of more effective DA techniques can result in substantial improvements across various tasks [49-51], such as Mixup [42] for image classification, DiffAugment [52] for GAN training, and Copy-Paste [53] for instance segmentation, among others. However, there has been a lack of investigation into the design of DA operations that specifically align with the requirements of sample-efficient visual RL. Only the early studies, RAD [10] and DrQ [4], conducted initial ablations and suggested that geometric transformations, exemplified by PadCrop, can achieve the best sample efficiency performance. More details regarding the previous DA design are in Appendix A.2. To achieve more precise design of DA operations, our study delves into an in-depth analysis of the essential DA attributes required for sample-efficient visual RL. Based on this analysis, we propose two actionable guidelines along with corresponding specific improvement strategies to enhance the effectiveness of DA methods.
Multi-Type DA Fusion Methods. The commonly employed multi-type DA fusion schemes can be categorized into three types: composing-based fusion [54, 55], mixing-based fusion [46, 48] and sampling-based fusion [56, 45]. Due to space constraints, we provide an expanded exposition of previous fusion schemes and their illustrations in Appendix A.3. While these fusion methods have proven effective in improving training accuracy and robustness in supervised and self-supervised learning [44, 18, 20, 57], our experiments in Section 3.2 indicate that they cannot result in improved sample efficiency in visual RL. Our proposed RL-tailored fusion scheme, CycAug, adopts individual operations during each update to prevent excessive increases in the hardness level, in contrast to composition-based and mixing-based fusion schemes [54, 55, 46, 19]. Additionally, CycAug performs periodic cycles of different DA operations at predetermined intervals, avoiding the instability caused by frequent switching of DA types between each update, unlike the sampling-based fusion approach [56, 45]. This way, CycAug can benefit from the type diversity brought by different operations while avoiding the aforementioned drawbacks.
3 Rethinking the Essential Attributes of DA for Sample-Efficient Visual RL
In this section, we analyze the attributes of DA, focusing on hardness and diversity, and thoroughly investigate their impact on DA's effectiveness in achieving sample-efficient visual RL.
3.1 Unpacking the Attributes of DA
Hardness. Hardness is a model-dependent measure of distribution shift, which was initially defined as the ratio of performance drop on augmented test data for a model trained in clean data [18, 20]. Considering the RL paradigm, the definition of hardness can be adapted to:
$$
\text {H a r d n e s s} = \mathcal {R} (\pi , \mathcal {M}) / \mathcal {R} (\pi , \mathcal {M} ^ {\mathrm {a u g}}), \tag {1}
$$
where  $\pi$  denotes the policy trained in the original Markov Decision Process (MDP)  $\mathcal{M}$  without DA.  $\mathcal{R}(\pi, \mathcal{M})$  and  $\mathcal{R}(\pi, \mathcal{M}^{\mathrm{aug}})$  represent the average episode return of policy  $\pi$  achieved in  $\mathcal{M}$  and
$\mathcal{M}^{\mathrm{aug}}$ , respectively. The only difference between the original MDP  $\mathcal{M} = (\mathcal{O}, \mathcal{A}, P, R, \gamma, d_0)$  and the augmented MDP  $\mathcal{M}^{\mathrm{aug}} = (\mathcal{O}^{\mathrm{aug}}, \mathcal{A}, P, R, \gamma, d_0)$  lies in the observation space where  $\mathcal{O}^{\mathrm{aug}}$  is obtained by applying DA to  $\mathcal{O}$ . Consistent with previous studies [18, 20], we observe a strong linear positive correlation between the hardness level and strength of individual DA operations in visual RL (refer to Appendix B.1 for further details). This implies that precise control of the hardness level of DA can be achieved by adjusting the strength of individual transformations.
Diversity. Another critical attribute of DA is diversity, which quantifies the randomness (degrees of freedom) of the augmented data [20, 18]. Typically, DA's diversity can be achieved through the following three approaches that we will investigate separately in Section 3.2:  $\bullet$  Strength Diversity refers to the variability of strength values, and a wider range indicates greater diversity [18]. In our experiments, we manipulate the strength diversity by expanding the range of hardness values while maintaining a constant average hardness.  $\bullet$  Spatial Diversity denotes the inherent randomness in the geometric transformations [20]. For example, the spatial diversity of the Translate operation can be characterized by the number of possible directions allowed for translations.  $\bullet$  Type Diversity is widely employed in numerous state-of-the-art DA methods by combining multiple types of operations. Enhancing type diversity is widely recognized as an effective strategy to explore the potential of DA. We provide a visual illustration in Appendix B.2 to facilitate a better understanding of the three different categories of diversity.
3.2 Investigating the Impact of DA's Attributes on Sample Efficiency
This section aims to comprehensively evaluate how hardness, strength diversity, spatial diversity, and type diversity impact the effectiveness of DA in sample-efficient visual RL. To conduct this investigation, we introduce three individual DA operations, namely PadResize-HD, CropShift-HD and Translate-HD. These operations provide precise control over the hardness level through mean strength, the strength diversity level through the range of strength, and the spatial diversity level through the degree of freedom. Meanwhile, each individual operation possesses a fixed level of type diversity. Implementation details and example demonstrations can be found in the Appendix C.1. Following experiments are conducted in the DeepMind Control (DMC) suite [22], where we apply diverse DA operations across various tasks to robustly assess the essential attributes. For each given scenario, we perform 10 random runs with 10 testing evaluations per seed and report the interquartile mean (IQM) of these 10 evaluation episode returns [58]. Detailed setup is in Appendix C.2.
How Hardness Affects the Effectiveness of DA? We investigate the impact of hardness on individual DA by varying the level of hardness while maintaining a constant level of diversity. As illustrated in Figure 2, the agent's performance exhibits a steep decline with the increase of hardness. In additional, DA can even lead to a deleterious impact once the hardness level goes beyond a certain threshold. Compared with other domains such as supervised learning [19, 21], our experiments reveal that the training process of visual RL is significantly sensitive to the hardness level of DA.
Figure 2: The impact of hardness on individual DA effectiveness. We present the IQM scores of 6 distinct individual DA operations, with fixed diversity levels while varying their hardness level. The maximum environment interactions is limited to  $5 \times 10^{5}$  steps (consistently throughout this section). "Strength D" and "Spatial D" are abbreviations for Strength Diversity and Spatial Diversity. The gray line denotes the performance without DA. Further comparative results are in Appendix C.3.
Finding 1: Maintaining a low level of DA hardness is a crucial priority in visual RL, as even minor increases in hardness can have a significant negative impact on training performance.
How Strength Diversity Affects the Effectiveness of DA? Strength diversity is a readily modifiable attribute that can improve the overall diversity of DA, for instance, by enlarging the range of rotation permitted in Rotate. We manipulate the range of strength while maintaining spatial diversity and a constant average hardness/strength level. The trends depicted in Figure 3 suggest that a moderate level of strength diversity can yield the most favorable enhancement effect. Insufficient strength diversity fails to meet the randomness demand of RL training, particularly when the individual DA's spatial diversity is constrained. Meanwhile, excessive strength diversity can also lead to a drastic decrease in the DA effect. This is attributed to the high sensitivity of RL to hardness, as increasing strength diversity unavoidably amplifies the highest hardness of the DA.
Figure 3: The impact of strength diversity on individual DA effectiveness. Strength diversity is generated by defining five gradual ranges of strength with a fixed mean hardness level. During training, the specific strength applied to different batches is randomly sampled from the given range. For detailed settings regarding the strength ranges, please consult Appendix C.4.
Finding 2: In contrast to supervised learning and adversarial training scenarios [20, 18], unlimited increases in strength diversity can harm visual RL performance.
How Spatial Diversity Affects the Effectiveness of DA? The PadCrop operation (also named Random Shift in the original paper [4, 5]) has shown outstanding effectiveness in improving the sample efficiency of visual RL algorithms. The subsequent studies further revealed that this individual PadCrop operation can provide significant smoothness and randomness, thus preventing unfavorable self-overfitting [59, 60]. Compared to other DA operations such as Translate and Rotate, PadCrop exhibits greater diversity in spatial randomness. As such, spatial diversity is highly regarded as a crucial determinant of whether DA can effectively enhance the sample efficiency of visual RL. We evaluate the influence of spatial diversity on the DA effect by incrementally loosening the restriction on it while maintaining a consistent level of hardness. The experimental trends consistently indicate that spatial diversity plays a crucial role in achieving effective DA.
Figure 4: The impact of spatial diversity on individual DA effectiveness. We provide a thorough analysis of the correlation between sample efficiency and spatial diversity of 3 individual DA operations at varying levels of hardness. Training curves are provided in the Appendix C.5.
Finding 3: Spatial diversity is a crucial attribute to achieve effective DA. When designing individual DA techniques, we should make every effort to increase their spatial diversity.
How Type Diversity Affects the Effectiveness of DA? Multi-type DA fusion methods have been widely adopted as a paradigm for overcoming the limitations of individual DA operations. This implies that enhancing type diversity is an effective approach to improve the effectiveness of DA [46, 45]. In this part, we conduct a comprehensive evaluation of the impact of type diversity by testing three main fusion schemes with six different DA operations. Surprisingly, none of the fusion schemes outperformed their corresponding individual operations. This abnormal failure can be attributed to the increased data hardness caused by the complex transformations (in the case of composition-based and mixing-based fusion [46]) or to the dynamic fluctuations in the data distribution that result from frequent switching between different types of DA operations (in the case of sampling-based fusion [45, 55]). These adverse effects are further compounded when transferring the fusion schemes from supervised learning to RL, mainly due to the non-stationary nature of the RL training process and its high sensitivity to the hardness of the DA [61, 62].
Figure 5: The impact of combining two distinct DA operations using three widely employed fusion schemes. The reported performances are the average scores evaluated after  $1.5 \times 10^{6}$  interactions on Quadruped Run. The entries along the main axis correspond to using only one individual operation. Additional experimental results and analysis can be found in Appendix C.6 for further reference.
Finding 4: Applying multi-type DA to visual RL training following general schemes from computer vision fields [46, 55, 45] can actually lead to decreased performance. Effective multi-type DA fusion schemes for achieving sample efficiency in visual RL must take the data-sensitive and non-stationary nature of RL training into consideration.
Overall, the aforementioned findings provide two key insights for further harnessing the potential of DA in sample-efficient visual RL: • For individual DA operations, maintaining a minor level of hardness and maximizing its spatial diversity are two crucial measures to enhance the effectiveness of DA. On the contrary, indiscriminately increasing strength diversity may not always be a prudent approach, as it inevitably amplifies the highest level of hardness. • For multi-type DA fusion schemes, exploring RL-tailored fusion schemes that leverage the advantages of diverse DA types is a promising approach to maximize the effectiveness of DA. However, adaptive fusion schemes must avoid introducing extra hardness of DA and ensure data stability during training.
4 Methodologies for Enhancing the Effectiveness of DA
Drawing on the insights gained in the previous rethinking, this section presents our corresponding improvement measures. Firstly, we propose a novel individual DA operation, Random PadResize (Rand PR), that ensures slight hardness by preserving content information while providing ample spatial diversity through random scaling and location. Secondly, we introduce Cycling Augmentation (CycAug), a multi-type DA fusion scheme tailored to the unstable nature of RL training.
4.1 Random PadResize (Rand PR): A Diverse yet Hardness-Minimal DA Operation
As recommended in the previous section, an individual DA operation for enhancing the sample efficiency of visual RL should provide as ample spatial diversity as possible while ensuring a low level of hardness. This insight reveals the key factors behind the previous impressive success of PadCrop (Shift) augmentation [10, 4, 5]. It offers greater spatial diversity than common Translate and Rotate operations due to its increased degree of freedom in shift, while also introducing lower augmentation hardness than Cutout and Color Jitter. However, the crop operation in PadCrop
inevitably leads to a loss of observation information. Although such loss may not have severe consequences in the DM Control suite [22] due to the agent always being in the center of the image with fixed camera position [59], it can lead to high hardness in real-world scenarios where task-relevant information near the observation boundaries will be reduced.