SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: (Left) Illustration of the Random PadResize operation. The four augmented observations exhibit substantial spatial diversity through differentiated scaling ratios and locations, while fully retaining the observation information. (Right) The hardness comparison between PadResize and PadCrop. We train 10 agents without DA in the Cartpole Balance task and evaluate their performance on augmented observations generated by PadCrop and PadResize. The results indicate that PadResize achieves lower hardness. For fairness, we set pad=4 in PadCrop and strength=16 in PadResize.
Motivated by aforementioned weakness of the original PadCrop operation, we introduce a modified version, Random PadResize (Rand PR), which offers increased diversity while minimizing augmentation hardness. As illustrated in Figure 6 (Left), Rand PR randomly adds a specified number of pixels in each direction of the image and then resizes it back to the original size. The sum of pixels padded in each direction (i.e., top, bottom, left, and right) is defined as the strength of this operation. Rand PR offers sufficient spatial diversity through its large degrees of freedom in scaling ratio and content location while avoiding the high hardness introduced by information reduction. The comparative experiment conducted in the Cartpole Balance task demonstrated that our Rand PR can achieve lower hardness than the original PadCrop operation, as shown in Figure 6 (Right).
4.2 Cycling Augmentation (CycAug): A Multi-Type DA Scheme Tailored to RL
As demonstrated in Section 3.2, generic multi-type DA fusion paradigms utilized in other fields cannot improve the sample efficiency of visual RL algorithms. Consequently, our second improvement measure focuses on devising a multi-type DA fusion scheme tailored for visual RL tasks.
Figure 7: Illustration of the CycAug scheme: (a) The CycAug pipeline consists of a simple workflow optimized for the RL objective, without the need for complex auxiliary representation tasks. (b) CycAug achieves multi-type DA fusion by cyclically switching between different types of operations.
We argue that the failure of generic multi-type DA fusion methods is mainly due to the struggle nature of RL training, which makes it highly sensitive to both the hardness and distribution evolving of the training data. Considering the inherent instability of the RL training process, we provide two recommendations: (1) to apply only one type of individual operation in each update to avoid introducing additional hardness, and (2) to appropriately reduce the frequency of switching between different operations to maintain the data distribution consistency during training. Based on these recommendations, we propose a multi-type DA fusion scheme specifically tailored for RL training, called Cycling Augmentation (CycAug). CycAug enables visual RL to benefit from type diversity while ensuring the adequate stability during the learning process. Specifically, CycAug applies individual operations during each update and cyclically switches between different types of operations after a certain number of steps. In addition, the experiments in Section 3.2 indicate that the efficacy of individual operations strongly influences the performance of their fusion. Consequently, in this paper, we adopt PadCrop and Random PR as two individual DA components for CycAug.
In this section, we evaluate the effectiveness of Rand PR and CycAug in enhancing the sample efficiency of visual RL algorithms. We conduct extensive experiments on both the DeepMind control suite [22] and the challenging autonomous driving simulator, CARLA [23] The code is accessible at https://github.com/Guozheng-Ma/CycAug. Furthermore, we provide empirical evidence to demonstrate the effectiveness of CycAug in ensuring training stability, as well as its scalability across an expanded range of augmentation quantities and types destined for fusion.
5.1 Evaluation on DeepMind Control Suite
Setup. We first evaluate our methods on continuous control tasks in the DM Control suite. Concretely, we integrate Rand PR and CycAug into the training procedure and network architecture of DrQ-V2 [5], while modifying its DA operation. We also compare our results with A-LIX [59], which achieved superior efficiency on this benchmark by using adaptive regularization on the encoder's gradients. Additionally, we include the performance of the original DrQ [4], CURL [16], and SAC [63] to demonstrate the significant improvement in sample efficiency achieved by our methods. For evaluation, we have chosen a collection of 12 challenging tasks from the DMC that showcase various difficulties such as complex dynamics, sparse rewards, and demanding exploration requirements. To assess sample efficiency, we limit the maximum number of training frames to  $1.5 \times 10^{6}$ , which is half of the value employed in DrQ-V2. The cycling interval of CycAug is set to  $10^{5}$  steps, i.e.,  $2 \times 10^{5}$  frames with 2 action repeat. More setup details are placed in Appendix D.1 due to page limit.
Table 1: Evaluation of Sample Efficiency on the DeepMind Control Suite. We report the average episode return over 10 random seeds and 10 evaluation runs after training for 1.5M frames. The results of previous methods are sourced from the A-LIX [59] report.
Tasks	SAC	CURL	DrQ	DrQ-V2	A-LIX	Rand PR	CycAug
Acrobot Swingup	8 ± 9	6 ± 5	24 ± 27	256 ± 47	270 ± 99	240 ± 37	274 ± 93
Cartpole Swingup Sparse	118 ± 233	479 ± 329	318 ± 389	485 ± 396	718 ± 250	549 ± 125	682 ± 297
Cheetah Run	9 ± 8	507 ± 114	788 ± 59	792 ± 29	806 ± 78	745 ± 28	799 ± 62
Finger Turn Easy	190 ± 137	297 ± 150	199 ± 132	854 ± 73	546 ± 101	846 ± 112	889 ± 97
Finger Turn Hard	79 ± 73	174 ± 106	100 ± 63	491 ± 182	587 ± 109	619 ± 289	829 ± 217
Hopper Hop	0 ± 0	184 ± 127	268 ± 91	198 ± 102	287 ± 48	285 ± 122	344 ± 154
Quadruped Run	68 ± 72	164 ± 91	129 ± 97	419 ± 204	528 ± 107	543 ± 131	634 ± 151
Quadruped Walk	75 ± 65	134 ± 53	144 ± 149	591 ± 256	776 ± 37	689 ± 99	812 ± 93
Reach Duplo	1 ± 1	8 ± 10	8 ± 12	220 ± 7	212 ± 3	217 ± 13	225 ± 5
Reacher Easy	52 ± 64	707 ± 142	600 ± 201	971 ± 4	887 ± 19	965 ± 8	971 ± 9
Reacher Hard	3 ± 2	463 ± 196	320 ± 233	727 ± 172	720 ± 83	725 ± 91	697 ± 179
Walker Run	26 ± 4	379 ± 234	474 ± 148	571 ± 276	691 ± 10	642 ± 141	702 ± 219
Average score	52.28	291.73	281.03	547.96	585.67	588.75	654.83
Figure 8: Training Curves of 4 Representative Challenging Tasks in DMC: Finger Turn Hard, Hopper Hop, Quadruped Run and Walker Run. The curves of DrQ-V2 are adopted from its original paper.
Results. The evaluation results are exhibited in Table 1, along with the training curves of four representative tasks, presented in Figure 8. Overall, Rand PR achieves competitive sample efficiency compared to the original PadCrop operation, while CycAug outperforms all previous methods in both efficiency and final performance by a significant margin. In particular, CycAug demonstrates more
substantial performance improvements in several challenging tasks, including Finger Turn Hard with sparse rewards and Quadruped Run with a large action space. This confirms that cycling-based fusion of multi-type DA operations effectively mitigates catastrophic self-overfitting, which prior works identified as a critical limitation to visual RL performance [59, 64, 65, 60].
5.2 Evaluation on Autonomous Driving in CARLA
Setup. We conduct a second round of evaluation on more challenging tasks using the autonomous driving simulator CARLA [23]. We adopt the reward function and task setting from prior work [66], where the objective of the agent is to drive along a winding road for as long as possible within limited time-steps while avoiding collisions with other vehicles. We set the maximum number of allowed environment interactions to  $2 \times 10^{5}$  steps, and configure the CycAug cycling interval to be  $2 \times 10^{4}$  steps. To demonstrate the efficacy of our proposed augmentation methods, we compare Rand PR and CycAug against the original DrQ-V2 with random shift (PadCrop) and the baseline without DA. We include a detailed account of the experimental setup and hyperparameters in Appendix D.2.
Table 2: Evaluation of sample efficiency on CARLA with 4 different weathers. The average episode returns are averaged over 5 seeds and 20 evaluation runs after 100k and 200k training steps.
Weather	100k Steps	200k Steps
w/o DA	DrQ-V2	Rand PR	CycAug	w/o DA	DrQ-V2	Rand PR	CycAug
Default	62.6 ± 15	94.7 ± 34	99.5 ± 13	142.9 ± 12	90.3 ± 29	213.7 ± 27	232.9 ± 8	251.0 ± 15
WetNoon	65.4 ± 11	106.8 ± 12	118.8 ± 20	147.5 ± 9	106.5 ± 26	208.5 ± 13	215.4 ± 18	263.6 ± 16
SoftRainNoon	46.7 ± 30	93.9 ± 21	121.2 ± 7	140.7 ± 13	90.6 ± 62	219.6 ± 24	221.9 ± 40	286.9 ± 39
HardRainSunset	52.2 ± 35	103.5 ± 7	103.9 ± 18	142.1 ± 15	82.7 ± 60	229.4 ± 16	240.5 ± 15	277.2 ± 37
Average Score	56.7	99.7	110.8	143.3 (+43.7%)	92.5	217.8	227.7	269.7 (+23.8%)
Figure 9: Training Curves of CycAug, Rand PR, original DrQ-V2 [5] and the baseline without DA across 4 CARLA environments. The model architecture follows that of DrQ-V2, while we focus on manipulating the DA operations. The line plots denote the average episode returns across 5 seeds.
Results. The evaluation results presented in Table 2 and Figure 9 further validate the efficacy of our proposed improvements in enhancing DA for more sample-efficient visual RL. Different weather conditions not only provide a rich set of realistic observations but also introduce diverse dynamics within the environments. In general, Rand PR outperforms the original DrQ-V2 with PadCrop as DA, indicating that preserving boundary information can effectively reduce the hardness of DA and thus improves sample efficiency in realistic environments. Furthermore, CycAug outperforms the previous SOTA algorithm by substantial margins  $23.8\%$  in final performance and  $43.7\%$  in low data regime.
5.3 Ablation Study and Detailed Analysis.
Training Stability. RL training is highly sensitive to data stability [67]; hence, when fusing various DA operations to enhance diversity, it is imperative to minimize frequent distribution shifts to ensure stability. To quantitatively assess training stability, we measure the  $Q$  Variance and  $Q$ -Target Variance during the training process, adhering to the experimental methodology outlined by [68]. The results presented in Figure 10 demonstrate that CycAug can optimally preserve training stability in comparison to other fusion approaches.
Expanded DA Quantities and Types. We primarily demonstrate the performance of CycAug when fusing PadCrop and Rand PR as components. However, as a multi-type DA fusion scheme, CycAug
Figure 10: Comparison of Q Variance and Q-Target Variance during the training process with CycAug and other multi-type DA fusion schemes. Variance is consistently assessed through 4 forward passes, employing random data augmentation on identical observations. CycAug demonstrates notably superior data stability, which is crucial during the training process of visual RL.
can integrate a wider variety of DA operations in greater quantities, provided these DA operations are individually effective. We illustrate the scalability of CycAug in Figure 11. Three-component CycAug utilizing PC, PR, and CS attains superior sample efficiency versus the dual-component CycAug with PC and PR, exhibiting the capacity for additional expansion of CycAug.
Figure 11: Expanded DA components integrated by CycAug. (Left) More Types. Employing CycAug to fuse the commonly employed DA approach of PadCrop (PC) with other DA operations reveals varied performance outcomes. (Right) More Quantities. The effect of further integrating the original CycAug (PC+PR) with the third DA operation depends on the effectiveness of using that DA alone. PC+PR+CS surpasses PC+PR, highlighting CycAug's potential for further enhancement.
In conclusion, this work investigates the critical DA attributes for improving sample efficiency in visual RL and proposes two methods to enhance its effectiveness. We find that both spatial diversity and slight hardness are crucial for individual DA operations, and introduce Random PadResize (Rand PR) as a new transformation that satisfies these attributes. We also propose a multi-type DA fusion scheme, referred to as Cycling Augmentation (CycAug), which employs a cyclic application of different DA operations to enhance type diversity while ensuring data distribution consistency for stability. The evaluations on the DeepMind Control suite and CARLA demonstrate the superior sample efficiency of CycAug with Rand PR as a key component, surpassing previous SOTA methods. Furthermore, this work demonstrates the potential of enhancing sample-efficient visual RL through more effective DA operations, without modifying the underlying RL algorithm or network architecture. Therefore, we advocate for a greater emphasis on data-centric methods in the visual RL community.
Limitations. We investigate the essential attributes of DA for sample-efficient visual RL. However, further research is needed to understand how these attributes impact visual RL training fundamentally.
Acknowledgements. This work is supported by STI 2030—Major Projects (No. 2021ZD0201405). We thank Yunjie Su, Zixuan Liu, and Zexin Li for their valuable suggestions and collaboration. We sincerely appreciate the time and effort invested by the anonymous reviewers in evaluating our work, and are grateful for their valuable and insightful feedback.
[59] Edoardo Cetin, Philip J Ball, Stephen Roberts, and Oya Celiktutan. Stabilizing off-policy deep reinforcement learning from pixels. In International Conference on Machine Learning, pages 2784-2810. PMLR, 2022.
[60] Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pages 16828-16847. PMLR, 2022.
[61] Byungchan Ko and Jungseul Ok. Efficient scheduling of data augmentation for deep reinforcement learning. arXiv preprint arXiv:2206.00518, 2022.
[62] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Anima Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. arXiv preprint arXiv:2106.09678, 2021.
[63] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018.
[64] Jiaxian Guo, Mingming Gong, and Dacheng Tao. A relational intervention approach for unsupervised dynamics generalization in model-based reinforcement learning. In International Conference on Learning Representations, 2022.
[65] Ziheng Wang and Justin Sirignano. Global convergence of the ode limit for online actor-critic algorithms in reinforcement learning. arXiv preprint arXiv:2108.08655, 2021.
[66] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742, 2020.
[67] Evgenii Nikishin, Pavel Izmailov, Ben Athiwaratkun, Dmitrii Podoprikhin, Timur Garipov, Pavel Shvechikov, Dmitry Vetrov, and Andrew Gordon Wilson. Improving stability in deep reinforcement learning with weight averaging. In Uncertainty in artificial intelligence workshop on uncertainty in Deep learning, 2018.
[68] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in neural information processing systems, 34:3680-3693, 2021.
Supplementary Material for Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning
In this material, we provide additional information on previous related work and background knowledge, as well as detailed experimental setup and results. Appendix A provides an extensive review of relevant literature, offering a comprehensive background that helps contextualize the significance and positioning of this paper. In Appendix B, we present a comprehensive analysis of decoupling and accurately controlling the hardness and diversity of DA, forming the basis for our experimental framework and further investigations. In Appendix C and Appendix D, we present comprehensive information regarding the experimental setup, hyperparameters, and detailed results for the rethinking and evaluation parts, respectively.
A.1 What Characterizes Good DA
DA has become a ubiquitous technique in the whole deep learning community [42, 46, 43]. After decades of rapid development, there has been recently increasing attention focused on reevaluating which properties contribute to the effectiveness of DA [18-21, 47]. Rethinking "what makes good DA" not only enhances our understanding of how DA works but also guides the development of more effective techniques. [20] first introduced two easy-to-compute metrics, affinity (the inverse of hardness) and diversity, to decompose the attributes of DA and revealed that augmentation performance is determined by both affinity and diversity. Following this analytical framework, [18] demonstrated that diversity has the capacity to improve both accuracy and robustness. Contrarily, hardness can enhance robustness at the expense of accuracy, up to a certain threshold, beyond which both accuracy and robustness may be compromised. This paper conducts a comprehensive analysis to investigate the specific requisites of hardness and diversity in DA for sample-efficient visual RL. It unveils fundamental distinctions compared to other domains: (1) Visual RL training is highly sensitive to improvements in hardness, which is distinct from the fact that a certain level of hardness is necessary to improve robustness in supervised learning scenarios [19, 18]. (2) Increasing strength diversity and type diversity without considering the data-sensitive nature of RL training may not effectively enhance DA's performance in visual RL.
Apart from examining the essential attributes of good DA methods in terms of hardness and diversity, there have been studies dedicated to identifying additional indicators that can assess the effectiveness of DA in specific scenarios. In natural language processing (NLP) tasks, DND [21] recommends that good DA operations should generate diverse and challenging samples to provide informative training signals, while preserving the semantics of the original samples. In knowledge distillation (KD) scenarios, [47] conducted a statistical investigation and found that an effective DA scheme should minimize the cross-entropy variance between the teacher and student models. Exploring the RL-tailored metrics to assess the effectiveness of DA in sample-efficient visual RL represents a promising direction for future research.
A.2 DA Design in Sample-Efficient Visual RL
There are two primary works that compare the impact of different DA operations on the sample efficiency of visual RL. DrQ [4] evaluates the effectiveness of 6 popular augmentation techniques, including random shifts, vertical and horizontal flips, random rotations, cutouts and intensity jittering. RAD [10] investigates and ablates crop, translate, window, flip, rotate, grayscale, cutout, cutout-color, random convolution, and color jitter augmentations. Following the classification criteria for DA operations in survey [3], the conclusions derived from the analysis of DrQ and RAD can be summarized as follows: (1) The photometric transformations, such as color jitter, and the random erasing operations, such as cutout, are generally found to be ineffective for sample-efficient visual RL. (2) Geometric transformations, represented by crop and shift operations, exhibit the strongest efficacy in significantly improving the sample efficiency. Despite the recognized effectiveness of
geometric transformations, there remains a scarcity of research dedicated to exploring the fine-tuning of operation details and parameters in DA. This study aims to derive actionable guidelines for the DA design by investigating the essential attributes of DA required for achieving sample-efficient visual RL. Based on our thorough analysis, we propose a more effective DA operation called Random PadResize (Rand PR). Compared with the shift operation in DrQ [4] and the crop operation in RAD [10], Rand PR exhibits higher spatial diversity by introducing randomness not only in positioning/translation but also in the scaling factor. Meanwhile, Rand PR effectively mitigates the DA hardness by preserving edge information in observations.
Additionally, [8] provides evidence supporting the assertion that employing DA alone achieves higher sample efficiency compared to jointly learning additional self-supervised losses. To substantiate this claim, the authors conduct ablation experiments to analyze the distinct effects brought about by various DA. The observed phenomena in the DM Control tasks can be explained by the conclusions drawn from our study: In the case of the crop operation, where large-sized original rendered observations are randomly cropped to  $84 \times 84$  images as augmented data, it is evident that the augmentation effect is noticeably superior when the original size is  $100 \times 100$  as opposed to  $88 \times 88$ . This can be attributed to the fact that in DM Control tasks, where edge information is largely task-irrelevant, a slightly larger original size can promote higher spatial diversity, thereby improving the DA effectiveness.
A.3 Multi-Type DA Fusion Schemes
Beyond individual transformations, recent research has shifted towards combining different DA operations to achieve a more diverse and powerful augmentation, which is termed as multi-type DA fusion. The common fusion schemes can be categorized into three types: (1) Composing-based fusion is a fundamental fusion scheme that combines different types of DA operations sequentially, either in a fixed or random order [54, 55]. (2) Sampling-based fusion applies a single augmentation to each image, with the DA operation and strength (uniformly) sampled from the corresponding spaces [56]. TrivialAugment [45] serves as a strong representative example of this scheme. (3) Mixing-based fusion, exemplified by AugMix [46], randomly combines diverse augmented data generated by different operations to produce the final augmented data [19, 48]. Figure 12 illustrates the visual representation of the flow process for these three fusion schemes.
Figure 12: Illustration of composing-based, sampling-based and mixing-based fusion schemes.
Automatic DA Combined with Multi-Type DA Fusion The aforementioned fusion methods can be further improved by combined with automatic DA techniques, such as adversarial training (AugMax [19]) and hyper-parameter search (AutoAugment [44]). However, automatic DA comes with additional computational costs, which can be unfriendly to RL training that is already computationally expensive. Moreover, recent studies indicate that increased type diversity and randomness appear to be crucial attributes for superior DA performance [45, 18, 55]. Therefore, this paper focuses on the random fusion of multi-type DA.
B Hardness and Diversity in Visual RL
In this section, we provide supplementary details on the decomposition of DA into hardness and diversity components. We demonstrate the existing linear correlation between the hardness level and transformation strength of individual DA operations; and introduce the specific methodologies for manipulating the strength diversity and spatial diversity of DA.
B.1 The Relationship between Hardness and Strength
Hardness was initially introduced as a metric to quantify the data distribution shift caused by DA [20]. It is defined as the ratio of performance drop on augmented test data for a model trained on clean data. In contrast to supervised learning, where DA is employed to enhance the robustness and generalization of models, in many visual RL scenarios, it is simply not possible to obtain effective policies without DA. Hence, except for exceptionally simple tasks like Cartpole Balance, the initial definition is not applicable for measuring the hardness level of DA operations.
Figure 13: Relationship between Hardness and Strength of DA