Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning
Authors: Guozheng Ma
Linrui Zhang
Haoyu Wang
Lu Li
Zilin Wang
Zhen Wang
Li Shen
Xueqian Wang
Dacheng Tao
Venue: NeurIPS 2023 poster (2023)
Keywords: Data Augmentation
Visual Reinforcement Learning
Sample Efficiency
TL;DR: This paper provides a comprehensive empirical study to investigate the crucial properties of effective DA and explore how to fully harness its potential to further enhance sample-efficient visual RL.

Abstract: Data augmentation (DA) is a crucial technique for enhancing the sample efficiency of visual reinforcement learning (RL) algorithms.
Notably, employing simple observation transformations alone can yield outstanding performance without extra auxiliary representation tasks or pre-trained encoders. However, it remains unclear which attributes of DA account for its effectiveness in achieving sample-efficient visual RL. To investigate this issue and further explore the potential of DA, this work conducts comprehensive experiments to assess the impact of DA's attributes on its efficacy and provides the following insights and improvements: (1) For individual DA operations, we reveal that both ample spatial diversity and slight hardness are indispensable. Building on this finding, we introduce Random PadResize (Rand PR), a new DA operation that offers abundant spatial diversity with minimal hardness. (2) For multi-type DA fusion schemes, the increased DA hardness and unstable data distribution result in the current fusion schemes being unable to achieve higher sample efficiency than their corresponding individual operations. Taking the non-stationary nature of RL into account, we propose a RL-tailored multi-type DA fusion scheme called Cycling Augmentation (CycAug), which performs periodic cycles of different DA operations to increase type diversity while maintaining data distribution consistency. Extensive evaluations on the DeepMind Control suite and CARLA driving simulator demonstrate that our methods achieve superior sample efficiency compared with the prior state-of-the-art methods.

Learning Better with Less: Effective Augmentation for Sample-Eficiien Visual Reinforcement Learning
Guozheng Ma
Linrui Zhang'
Haoyu Wang',
Lu Li'
Zilin Wang'
Zhen Wang" Li Shen Xueqian Wang!* Dacheng Tao
Tsinghua University Thhe University of Sydney $JD Explore Academy
[mgz21, hwoyuwa22 11ll11,, sanga121)8mails tsinghua. edu.
zwan4121 21e22nnnni... momni..dddey.eee edu. au; wang. xqeez. xq@ez.tssnnguu. edu.cm cn [zhanglr.cg auto, wwt.shheelinnn..mmmmmmmmmm iaapheennddccchnngn dacheng. tao]]gmail.com
Abstract
Data augmentation (DA) is a crucial technique for enhancing the sample efficiency of visual reinforcement learning (RL) algorithms. Notably, employing simple observation transformations alone can yield outstanding performance without extra auxiliary representation tasks pre-trained encoders. However, it remains unclear which attributes of DA account for its effectiveness in achieving sumple-efficien visual RI. To investigate this issue and further explore the potential of DA, this work conducts comprehensive experiments to assess the impact of DA's attributes on its efficacy and provides the following insights and improvements: (I) For individual DA operations, we reveal that both ample spatial diversity and slight hardness are indispensable. Building on this finding, we introduce Random PadResize (Rand PR), new DA operation that offers abundant spatial diversity with minimal hardness. (2) For multi-rype DA fusion schemes, the increased DA hardness and unstable data distribution result in the current fusion schemes being unable to achieve higher sample efficiency than their corresponding individual operations. Taking the non-stationary nature of RI. into account, we propose RL-tailored multi-type DA fusion scheme called Cycling Augmentation (CycAug), which performs periodic cycles of different DA operations to increase type diversity while maintaining data distribution consistency. Extensive evaluations on the DeepMind Control suite and CARLA driving simulator demonstrate that our methods achieve superior sample efficiency compared with the prior state-of- of-the-art methods.
Introduction
Visual reinforcement learning (RL) has shown great potential in various domains, enabling socision-making directly from high-dimensional visual inputs [1]. However, the dual requirements of simultaneously learning compact state representations and optimizing task-specific policies lead to prohibitive sample complexity and massive environment interactions, hindering its practical deployment [2]. Enhancing sample efficiency is a critical and inevitable challenge faced by the entire visual RL community, and data augmentation (DA) has emerged
Walker Ron
Daess Angrestanie
Amazing LEgASANMARD saaplc oflicanszy Dazefiseo
remarkably effective ap-
proach tackle this issue [3-7]. As illustrated Figure i, even simple random shift transformations applied to input observations can lead to significant performance improvements in previously unsuccessful algo- rithm [5]. Moreover, it has been proven that DA alone can lead to more
Figure 1: Benefit of DA in visual RL.
*Corresponiigg authors: Li Shen and Xueqian Wang
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

efficient performance compared with meticulously crafting self-supervised learning tasks [8] or pre-traiming representation encoders with extra data [9]. The substantial effectiveness of DA in improving sample efficiency has established it as an essential and indispensable component in the vast majority of visual RI. methods [4,5, 5,7, 7, 10-12].
However, recent research has predominandly concentrated on integrating DA with other comple- mentary techniques to enhance performance [7, 13-17], while ignoring the opportunity to fully explore and harness the intrinsic potential of DA operations themselves. There is a notable absence of actionable guidelines for designing more effective DA operations specifically tailored to visual RL scenarios. Motivated by this imperative requirement, is crucial to undertake a thorough investigation and in in-depth analysis to explore a fundamental question:
Which attributes enable effective DA for achieving sample-efficient visual RL?
Hence, this study begins with comprehensive experiments in order to assess the essential attributes of DA for enhancing sample efficiency in visual RL. Specifically, we unpack the attributes of DA operations in terms of hardness and diversity, which are two main dimensions that previous studies leveraged to deconstruct DA in other domains [18-21]. The extensive experiments provide strong support for the following key findings, which highlight the distinctive requirements of visual RL: The training performance of visual RL is highly sensitive to the increase of DA's hardness. Sufficient spatial diversity is an indispensable attribute for achieving effective DA. Unlimited increases in strength diversity can harm visual RL performance. Despite the increased type diversity, naively applying multi-type DA to visual RL training can lead to decreased performance.
Drawing upon these findings, we propose actionable guidelines and corresponding strategies for enhancing the effectiveness of DA in visual RL. (1) For individual DA operations, we reveal that effective operations should exhibit sufficient spatial diversity with minimal hardness. Building on this insight, we introduce Random PadResize (Rand PR), new DA transformation that satisfies the essential attributes required for visual RL. Rand PR alleviates the hardness of DA by preventing the loss of information in observations, which is inevitable in the conventional PadCrop transformation that widely used in previous methods such as DrQ-v2 [5]. Additionally, it provides significant spatial diversity by randomly scaling and positioning the original content within the final input space. (2) For multi-type DA fusion schemes. it is imperative take the sata-sensi nature of RL training into account when designing appropriate fusion schemes in visual RL. Inspired by this guideline, we propose a RL--tallrred multi-type DA fusion scheme called Cycling Augmentation (CyeAug), which leverages the diversity benefits of multiple DAs by cyclically applying different transformations while mitigating the negative impact on training stability caused by the excessively frequent changes in data distribution. With Rand PR as a key component, CycAug achieves superior sample efficiency on various tasks of the DeepMind Control suite [22]. Moreover, in the challenging low data regime of CARLA [23], CycAug outperforms the previous SOTA algorithm by substantial margin of 43. 13.7%
In the end, our main contributions can be summarized as following threc-fold:
1. We conduct extensive experiments to investigate the essential DA attributes for achieving sample-efficient visual RL and highlight the unique requirements of visual RL for DA.
2. We present two actionable guidelines to further exploit the potential of DA, specifically focusing on individual DA operations and multi-type -tpee DA fusion schemes. Building upon these guidelines, we propose Rand PR and CycAug as corresponding improvement strategies.
3. CycAug, incorporating Rand PR as a key component, achieves state-of-the-art sample efficiency in extensive benchmark tasks on DM Control and CARLA.
2
Related Work
In this section, we provide a succinct overview of prior research conducted in sample-efficient visual RL, as well as DA methods utilized in the broader domain of deep learning.
2.1 Sample-Efficient Visual RL
Visual RL entails learning compact state representations from high-dimensional observations and optimizing task-specific policies at the same time, resulting in prohibitive sample complexity [2]. To enhance sample efficiency, the first widely adopted approach leveraging self-supervised learning to promote agent's representation learning. Specifically, various auxiliary tasks have been designed
2

---

in previous studies, including pixel or latent reconstruction [2, 14], future prediction [24, 7, 13] and contrastive learning for instance discrimination [16, 17, 15] or temporal discrimination [25- 28]. Another approach aimed at facilitating representation learning is to pre-train visual encoder that enables efficient adaptation to downstream tasks [29-34]. Additionally, a number of model- based methods explicitly construct a world model of the RL environment in either pixel or latent spaces to enable planning [35-41]. Orthogonal to these methods, DA has demonstrated astonishing effectiveness in improving the sample efficiency of visual RL algorithms [10, 4, 5J. Moreover, recent research has focused on effectively integrating DA with other representation learning methods to further enhance sample efficiency [7, 13, 16, 25, 14]. However, the intrinsic potential of DA operations has been inadequately explored. To address this research gap, this study conducts thorough experiments to investigate the fundamental attributes that contribute the effectiveness of DA and subsequently provide actionable guidelines for DA design. Furthermore, we illustrate that developing more effective DA methods alone can result in substantial improvement in sample efficiency.
2.2 Data Augmentation (DA)
Over the past decades, DA has achieved widespread recognition and adoption within the deep learning community [42-48]. For clarity, we classify previous studies DA into two distinct aspects: individual DA operations, which utilize single-type data transformations, and multi -tplii--tpeeeeee DA fusion methods, which combine diverse DA operations through specific fusion schemes.
Individual DA Operations. Numerous studies have shown that the adoption of more effective DA techniques can result in substantial improvements across various tasks [49-51], such as Mixup [42] for image classification, DiffAugment [52] for GAN training. and Copy-Paste [53] for instance segmentation, among others. However, there has been a lack of investigation into the design of DA operations that specifically align with the requirements of sample-efficient visual RL. Only the early studies, RAD [10] and DrQ [4], conducted initial ablations and suggested that geometric transformations, exemplified by PadCrop, can achieve the best sample efficiency performance. More details regarding the previous DA design are in Appendix A.2. To achieve more precise design of DA operations, our study delves into an in-depth analysis of the essential DA attributes required for sample-efficient visual RI. Based on this analysis, we propose two actionable guidelines along with corresponding specific improvement strategies to enhance the effectiveness of DA methods.
Multi-Type DA Fusion Methods. The commonly employed multi-type DA fusion schemes can be categorized into three types: composing-basee fusion [54, 55], mixing- baaed fusion [46, 48] and sampling-based fusion [56, 45]. Due to space constraints, we provide an expanded exposition of previous fusion schemes and their illustrations in Appendix A.3. While these fusion methods have proven effective in improving training accuracy and robustness in supervised and self- sule-supe learning [44, 18, 20, 57], our experiments in Section 3.2 indicate that they cannot result in improved sample efficiency in visual RI. Our proposed RL-tailored fusion scheme, CycAug, adopts individual operations during each update to prevent excessive increases in the hardness level, in contrast to composition-basse
and mixing-based fusion schemes [54, 55, 46, 19]. Additionally, CycAug performs periodic cycles of different DA operations at predetermined intervals, avoiding the instability caused by frequent switching of DA types between each update, unlike the sampling-base fusion approach [56, 45]. This way, CycAug can benefit from the type diversity brought by different operations while avoiding the aforementioned drawbacks.
3
Rethinking the Essential Attributes of DA for Sample-Efficien Visual RL
In this section, we analyze the attributes of DA, focusing on hardness and diversity, and thoroughly investigate their impact on DA's effectiveness in achieving sample-efficient visual RL.
3.1
Unpacking the Attributes of DA
Hardness. Hardness is a model-dependent measure of distribution shift, which was initially defined as the ratio of performance drop on augmented test data for a model trained in clean data [18, 20]. Considering the RI. paradigm, the definition of hardness can be adapted to:
(I) where æ denotes the policy trained in the original Markov Decision Process (MDP) M without DA. R MTM and R (T,, Maig)) represent the average episode return of policy π achieved in M and
Hardness = R (T, M)/M (TT, MMn)).
3

---

M respectively. The only difference between the original MDP M (O,A.,,RR du) and the augmented MDP MM (Oang A. P. R, 7, do) lies in the observation space where Oung is obtained by applying DA to O. Consistent with previous studies [18, 20], we observe a strong linear positive correlation between the hardness level and strength of individual DA operations in visual RI. (refer to Appendix B.1 for further details). This implies that precise control of the hardness level of DA can be achieved by adjusting the strength of individual transformations.
Diversity. Another critical attribute of DA is diversity, which quantifies the randomness (degrees of freedom) of the augmented data [20, 18]. Typically, DA's diversity can be achieved through the following three approaches that we will investigate separately in Section 3.2: Strength Diversity refers to the variability of strength values, and wider range indicates greater diversity [18]. In our experiments, we manipulate the strength diversity by expanding the range of hardness values while maintaining a constant average hardness. Spatial Diversity denotes the inherent randomness in the geometric transformations [20]. For example, the spatial diversity of the Translate operation can be characterized by the number of possible directions allowed for translations. Type Diversity is widely employed in numerous taateoffthe-ar DA methods by combining multiple types of operations. Enhancing type diversity is widely recognized as an effective strategy to explore the potential of DA. We provide a visual illustration in Appendix B.2 to facilitate better understanding of the three different categories of diversity.
3.2 Investigating the Impact of DA's Attributes on Sample Efficiency
This section aims to comprehensively evaluate how hardness, strength diversity, spatial diversity, and type diversity impact the effectiveness of DA in sample-efficient visual RL. To conduct this investigation, we introduce three individual DA operations, namely PadResize-His CropShift- and Translate-HD. These operations provide precise control over the hardness level through mean strength, the strength diversity level through the range of strength, and the spatial diversity level through the degree of freedom. Meanwhile, each individual operation possesses a fixed level of type diversity. Implementation details and example demonstrations can be found in the Appendix C.I. Following experiments are conducted in the DeepMind Control (DMC) suite [22], where we apply diverse DA operations across various tasks to robustly assess the essential attributes. For each given scenario, we perform 10 random runs with 10 testing evaluations per seed and report the interquartile mean (IQM) of these 10 evaluation episode returns [58]. Detailed setup is in Appendix C.2.
How Hardness Affects the Effectiveness of DA? We investigate the impact of hardness on individual DA by varying the level of hardness while maintaining a constant level of diversity. As illustrated in Figure the agent's performance exhibits steep decline with the increase of hardness. In additional, DA can even lead to a deleterious impact once the hardness level goes beyond a certain threshold. Compared with other domains such as supervised learning [19, 21], our experiments reveal that the training process of visual RL is significantly sensitive to the hardness level of DA.
drughna Rib Streeels 6-2, Syoted Do:) druaht di 6D Streoelh 2-2, spoted b-i
Prdllcca HD dhrooggh B-1 Apacial D-2 Paddecca dirooghh b-1 Susial B-o
Tranda a HD Brooggh D-2 Spaial b-:)) Triada HDD chrosgth D-2 Spatal bo-k)
Cup Catch
Finger Spin
Cheetah Run
020
5000
Tpyoorte Return
Eyloodo Buzura
302
1200
Eydsode
282
200
Hardiwes Lovel
Handnwes Lovel
Hardneos Lovel
Figure 2: The impact of hardness on individual DA effectiveness. We present the IQM scores of 6 distinct individual DA operations, with fixed diversity levels while varying their hardness level. The maximum environment interactions is limited to x 10" steps (consistently throughout this section). "Strength D" and "Spatial D" are abbreviations for Strength Diversity and Spatial Diversity. The gray line denotes the performance without DA. Further comparative results are in Appendix C.3.
Finding 1: Maintaining low level of DA hardness is a crucial priority in visual RL, as even minor increases in hardness can have a significant negative impact on training performance.

---

How Strength Diversity Affects the Effectiveness of DA? Strength diversity is a readily modifiable attribute that can improve the overall diversity of DA, for instance, by enlarging the range rotation permitted in Rotate. We manipulate the range of strength while maintaining spatial diversity and constant average hardness/strength level. The trends depicted in Figure suggest that moderate level of strength diversity can yield the most favorable enhancement effect. Insufficient strength diversity fails to meet the randomness demand of RL training, particularly when the individual DA's spatial diversity is constrained. Meanwhile, excessive strength diversity can also lead to a drastic decrease the DA effect. This is attributed to the high sensitivity of RL to hardness, as increasing strength diversity unavoidably amplifies the highest hardness of the DA.
Cup Catch
Finger Spin
Cheelah Run
300
2.0
Return
1.0
203
%
Algisszlaioe
202
Asptsiniuuss Cosptan-tDD Dels
pocckannD-llrrttan Sputia Did)
26
Crppiaat Trgplart-iiitteddee (Dtteedaenn Spetial D-4)
Pelkeaiee Street Cuputa (t-4)
110
Meilbiuu antprartenned tpdlll traal
Strength Dicesitt
Strength Deveriity
Stongth Diversity
The impact of strength diversity individual DA effectiveness. Strength diversity is generated by defining five gradual ranges of strength with a fixed mean hardness level. During training, the specific strength applied to different batches is randomly sampled from the given range. For detailed settings regarding the strength ranges, please consult Appendix C.4.
Figure 3:
In contrast to supervised learning and adversarial training scenarios [20, 18], unlim- ited increases in strength diversity can harm visual RL performance.
Finding 2:
How Spatial Diversity Affects the Effectiveness of DA? The PadCrop operation (also named Random Shift in the original paper [4, 5]) has shown outstanding effectiveness in improving the sample efficiency visual RL algorithms. The subsequent studies further revealed that this individual PadCrop operation can provide significant smoothness and randomness, thus preventing unfavorable self-ovvriitting [59, 60]. Compared other DA operations such as Translate and Rotate, PadCrop exhibits greater diversity in spatial randomness. As such, spatial diversity is highly regarded as crucial determinant of whether DA can effectively enhance the sample efficiency of visual RL. We evaluate the influence of spatial diversity on the DA effect by incrementally loosening the restriction on while maintaining a consistent level of hardness. The experimental trends consistently indicate that spatial diversity plays crucial role in achieving effective DA.
Casch Creghiihhhhhhh
Oup Cynerrrrnntiv HDA
Chrobo Shi (T))
Chearsh Ben/PetReete 1779
Pinger Spantnnphhhteeee
Tlager Serr/helleeesstii
Plager Sprethhhneaeettt
Cherret Bart HoD
Spatial Diverstty
Spotial Diversity=2
Spotial Diversttydd
Spatial Diverstty-
Spotial Dii Limited
The impact of spatial diversity on individual DA effectiveness. We provide a thorough analysis of the correlation between sample efffciency and spatial diversity of 3 individual DA operations at varying levels of hardness. Training curves are provided in the Appendix C.5.
Figure 4:
Spatial diversity is crucial attribute achieve effective DA. When designing individual DA techniques, we should make every effort to increase their spatial diversity.
Finding 3:
5

---

How Type Diversity Affects the Effectiveness of DA? Multi-type DA fusion methods have been widely adopted a paradigm for overcoming the limitations of individual DA operations. This implies that enhancing type diversity is an effective approach to improve the effectiveness of DA [46, 45]. In this part, we conduct comprehensive evaluation of the impact of type diversity by testing three main fusion schemes with six different DA operations. Surprisingly, none of the fusion schemes outperformed their corresponding individual operations. This abnormal failure can be attributed to the increased data hardness caused by the complex transformations (in the case of composition-aan bbaed and mixing-based fusion [46]) or to the dynamic fluctuations in the data distribution that result from frequent switching between different types of DA operations (in the case of sampling- based fusion [45, 55]). These adverse effects are further compounded when transferring the fusion schemes from supervised learning to RL, mainly due to the non-stationary nature of the RL training process and its high sensitivity to the hardness of the DA [61, 62].
Composing- based Fusion
Sampling-ber -based Fusion
Mixing-base -aaing-baaeddd Fusion
TedCrop
537.3
504.7
461.7
354.0
419.6
350.7
TC
537.9
525.7
420.5
487.7
452.8
351.7
PC
537-9
405.1
393.81
374.2
415.4
334.4
Tadieeeto
504.7
543.3
4.00.2
399.2
409.0
309 7
PR
525.7
5/3
475.8
697.1
491
357.3
PR
100.1
643.3
413
300
401.3
331 7
400
Shut
451.7
445.8
463.1
450.8
470.8
446.6
417.5
208.0
391.8
413.1
445.0
360
400.4
336
Dankkt
354.3
356.2
382.5
437.3
446.6
483.5
71
374.2
368.4
366.3
341.3
800
419.5
420.5
463.1
962.5
377.5
263.8
492.8
421.6
417.5
430.1
377.5
256.7
112
413.4
401.3
400.4
341.3
377.3
188.1
BandComm
350.7
369.7
331.5
302.8
281.
290.0
Ce
251.77
357.1
290.0
358.0
256.7
258.0
C9
3354.4
351.7
339.
312.2
288.1
209.0
200
Figure 5: The impact of combining two distinct DA operations using three widely employed fusion schemes. The reported performances are the average scores evaluated after 1.5 x 10 interactions on Quadruped Run. The entries along the main axis correspond using only one individual operation. Additional experimental results and analysis can be found in Appendix C.6 for further reference.
Finding 4: Applying multi-type DA to visual RL training following general schemes from computer vision fields [46, 55, 45] can actually lead to decreased performance. Effective multi- type DA fusion schemes for achieving sample efficiency in visual RL must take the data-sensitive and non-stationary nature of RI. training into consideration.
Overall, the aforementioned findings provide two key insights for further harnessing the potential of DA in sample-efficient visual RI:: For individual DA operations, maintaining a minor level of hardness and maximizing its spatial diversity are two crucial measures to enhance the effectiveness of DA. On the contrary, indiscriminately increasing strength diversity may not always be a prudent approach, as inevitably amplifies the highest level of hardness. For multi-type DA fusion schemes, exploring RL-tailored fusion schemes that leverage the advantages of diverse DA types is promising approach to maximize the effectiveness of DA. However, adaptive fusion schemes must avoid introducing extra hardness of DA and ensure data stability during training.
4 Methodologies for Enhancing the Effectiveness of DA
Drawing on the insights gained in the previous rethinking, this section presents our corresponding improvement measures. Firstly, we propose a novel individual DA operation, Random PadResize (Rand PR), that ensures slight hardness by preserving content information while providing ample spatial diversity through random scaling and location. Secondly, we introduce Cycling Augmentation (CycAug), multi-type DA fusion scheme tailored to the unstable nature of RL training.
4.1 Random PadResize (Rand PR): A Diverse yet Hardness-Mininat DA Operation
As recommended in the previous section, an individual DA operation for enhancing the sample efficiency of visual RL should provide as ample spatial diversity as possible while ensuring a low level of hardness. This insight reveals the key factors behind the previous impressive success of PadCrop (Shift) augmentation [10, 4, 5]. It offers greater spatial diversity than common Translate and Rotate operations due to its increased degree of freedom in shift, while also introducing lower augmentation hardness than Cutout and Color Jitter. However, the crop operation in PadCrop

---

inevitably leads to a loss of observation information. Although such loss may not have severe consequences in the DM Control suite [22] due the agent always being in the center of the image with fixed camera position [59], it can lead to high hardness in real- wol-world scenarios where task-relevant information near the observation boundaries will be reduced.
Cartpole Balance
Sufficient Spatial Diversity, Complete Information Retention
Pad
Resize
Strength
Pad Left
Pad Right Pad Top Pad Bottom
Different Augmented Observations
Augieettttin
Figure 6: (Left) Illustration of the Random PadResize operation. The four augmented observations exhibit substantial spatial diversity through differentiated scaling ratios and locations, while fully retaining the observation information. (Right) The hardness comparison between PadResize and PadCrop. We train 10 agents without DA in the Cartpole Balance task and evaluate their performance on augmented observations generated by PadCrop and PadResize. The results indicate that PadResize achieves lower hardness. For fairness, we set pad=4 in PadCrop and strength=16 16 in PadResize.
Motivated by aforementioned weakness of the original PadCrop operation, we introduce modified version, Random PadResize (Rand PR), which offers increased diversity while minimizing augmenta- tion hardness. As illustrated in Figure 6 (Left), Rand PR randomly adds specified number of pixels in each direction of the image and then resizes it back to the original size. The sum of pixels padded in each direction (i.e., top, bottom, left, and right) defined as the strength of this operation. Rand PR offers sufficient spatial diversity through its large degrees of freedom in scaling ratio and content location while avoiding the high hardness introduced by information reduction. The comparative experiment conducted in the Cartpole Balance task demonstrated that our Rand PR can achieve lower hardness than the original PadCrop operation, as shown in Figure (Right).
4.2 Cycling Augmentation (CycAug): Multi-Type DA Scheme Tailored to RL
As demonstrated in Section 3.2, generic multi-type DA fusion paradigms utilized in other fields cannot improve the sample efficiency of visual RL algorithms. Consequently, our second improvement measure focuses on devising a multi-type DA fusion scheme tailored for visual RL tasks.
CycAug
Update
PadCrop
PadResize
PadCrop
Reinforcement Learning Objective
Agent Network
Replay Buffer
PadResize
Mini Batch
Steps
Overall Architecture CycAug
Cycle Training Steps
Figure 7: Illustration of the CycAug scheme: (a) The CycAug pipeline consists of simple workflow optimized for the RL objective, without the need for complex auxiliary representation tasks. (b) CycAug achieves multi-type DA fusion by cyclically switching between different types of operations.
We argue that the failure of generic multi-type DA fusion methods is mainly due to the struggle nature of RL training, which makes highly sensitive both the hardness and distribution evolving of the training data. Considering the inherent instability of the RL training process, we provide two recommendations: (1) apply only one type of individual operation in each update to avoid introducing additional hardness, and (2) to appropriately reduce the frequency of switching between different operations maintain the data distribution consistency during training. Based on these recommendations, we propose multi-type DA fusion scheme specifically tailored for RL training,
called Cycling Augmentation (CycAug). CycAug enables visual RL to benefit from type diversity while ensuring the adequate stability during the learning process. Specifically, CycAug applies individual operations during each update and cyclically switches between different types of operations after a certain number of steps. In addition, the experiments in Section 3.2 indicate that the efficacy of individual operations strongly influences the performance of their fusion. Consequently, this paper, we adopt PadCrop and Random PR as two individual DA components for CycAug.
7

---

Empirical Evaluation
In this section, we evaluate the effectiveness of Rand PR and CycAug in enhancing the sample efficiency of visual RL. algorithms. We conduct extensive experiments on both the DeepMind control suite [22] and the challenging autonomous driving simulator, CARLA [23] The code is accessible at https: //gi //gggttuuuu thub. www. /Guozheng- Ma/ Ha/Cycuug Furthermore, we provide empirical evidence to demonstrate the effectiveness of CycAug in ensuring training stability, as well as scalability across an expanded range of augmentation quantities and types destined for fusion.
5.1 Evaluation on DeepMind Control Suite
Setup. We first evaluate our methods on continuous control tasks in the DM Control suite. Concretely, we integrate Rand PR and CycAug into the training procedure and network architecture of DrQ-V2 [5], while modifying its DA operation. We also compare our results with A-LIX [59], which achieved superior efficiency this benchmark by using adaptive regularization the original DrQ [4]. CURL [16], and SAC [63]
the encoder's gradients.
Additionally, we include the performance to demonstrate the significant improvement in sample efficiency achieved by our methods. For evaluation, we have chosen a collection of 12 challenging tasks from the DMC that showcase various difficulties such as complex dynamics, sparse rewards, and demanding exploration requirements. To assess sample efficiency, we limit the maximum number of training frames to 1.5 x 10 which is half of the value employed in DrQ-V2. The cycling interval of CycAug is set to 10 steps, ie, 2 x 10 frames with 2 action repeat. More setup details are placed in Appendix D.I due to page limit.
Table I: Evaluation of Sample Efficiency on the DeepMind Control Suite. We report the average episode return over 10 random seeds and 10 evaluation runs after training for 1.5M frames. The results of previous methods are sourced from the A-LIX [59] report.
Tasks
SAC
CURL
DrQ
DrQ-V2
A-LIX
Rand PR
CycAng
Acrobot Swingup
8 4 9
6
24 + 27
256 + 47
270 + 99
240 + 37
274 + 93
Caripole Swingup Sparse
118 : 233
479 = 329
318 : 389
485 1 396
718 + 250
549 + 125
682 : 297
Cheetah Run
9 8
507 114
788 + 59
792 + 29
806 + 78
745 + 28
799 + 62
Finger Tum Easy
190 : 137
297 150
199 = 132
854 + 73
546 + 101
846 + 112
889 + 97
Finger Turn Hard
79 = 73
174 106
100 = 63
491 + 182
587 = 109
619 + 289
829 + 217
Hopper Hop
0 0
184 127
268 +
198 + 102
287 1: 48
285 + 122
344 1: 154
Quadruped Run
68 72
164
419 -
528 +
543
151
Quadruped Walk
75 65
134 53
144 + 149
501 + 256
776 1: 37
680 99
812
Reach Duplo
1+1
8 10
8 12
220 7
212 + 3
217 13
225 +
Reacher Easy
52 61
707 = 142
600 +: 201
971 +
887 + 19
965 8
971 +
Reacher Hard
3+2
463 196
320 + 233
727 1: 172
720 + 83
725 + 91
697 + 179
Walker Run
26 = 4
379 234
474 + 148
571 : 276
691 + 10
642 :: 141
702 + 219
Average score
52.28
291.73
281.03
547.96
585.67
588.75
654.83
Pinger Turn Hard
Hopper Hop
Quadruped Run
Walleer Run
800
Zoge
6.0
Rendine Builivaion
Insifine
250
Oylling Aegerortation
300
6/00
600
5:0
800
3000
110
200
50
0.3
0.3
1.5
1.2
1.5
3.0
Figure 8: Training Curves of 4 Representative Challenging Tasks in DMC: Finger Turn Hard, Hopper Hop, Quadruped Run and Walker Run. The curves of DrQ-V2 V2 are adopted from its original paper.
Results. The evaluation results are exhibited Table 1, along with the training curves of four representative tasks, presented in Figure 8. Overall, Rand PR achieves competitive sample efficiency compared to the original PadCrop operation, while CyeAug outperforms all previous methods in both efficiency and final performance by a significant margin. In particular, CycAug demonstrates more
8

---

substantial performance improvements in several challenging tasks, including Finger Turn Hard with sparse rewards and Quadruped Run with a large action space. This confirms that eycling- based fusion of multi-type DA operations effectively mitigates catastrophic self-overfitting, which prior works identified as critical limitation to visual RI. performance [59, 64, 65, 60].
5.2 Evaluation on Autonomous Driving in CARLA
Setup. We conduct a second round of evaluation on more challenging tasks using the autonomous driving simulator CARLA [23]. We adopt the reward function and task setting from prior work [66], where the objective of the agent is to drive along a winding road for as long possible within limited time-steps while avoiding collisions with other vehicles. We set the maximum number of allowed environment interactions to 2 x 10 steps, and configure the CycAug cycling interval to be 2 x 1 steps. To demonstrate the efficacy of our proposed augmentation methods, we compare Rand PR and CycAug against the original DrQ-V2 with random shift (PadCrop) and the baseline without DA. We include a detailed account of the experimental setup and hyperparameters in Appendix D.2.
e
Table Evaluation of sample efficiency on CARLA with 4 different weathers. The average episode returns are averaged over 5 seeds and 20 evaluation runs after 100k and 200k training steps.
100k Steps
200k Steps
Weather
w/o DA
DrQ-V2
Rand PR
CycAug
w/o DA
DrQ-V2
Rand PR
CycAug
Default
62.6 + 15
94.7 + 34
90.5 13
142.9 12
90.3 + 29
213.7 27
232.9 8
251.0 +: 15
WetNoon
65.4 11
106.8 +:
118.8 20
147.5 9
106.5 1:26
208.. :: 13
213.4 =
263.6 + 16
SoftRainNoon
46.7 1. 30
93.9 1 21
121.2 17
140.7 : 13
90.6 6.62
219.6
24
221.9
40
286.9 1: 39
HardRainSurset
52.2 + 35
103.5 + 7
103.9 18
142.1 + 15
82.7 60
220.4 + 16
240.5
15
277.2 + 37
Average Score
56.7
99.7
110.8
143.3 (-43.79)
92.5
217.8
227.7
269.7 (+23.8%)
Defoult
WetNoon
SoftRainNoon
HardReinSunset
Trme PeaCres BroAg)
Oyt lepramtntion
330
3.33
3.3
Ay Pon
33
2.5
0.0
Figure 9: Training Curves of CycAug, Rand PR, original D2Q [5] and the baseline without DA across 4 CARLA environments. The model architecture follows that of DrQ-V2, while we focus on manipulating the DA operations. The line plots denote the average episode returns across 5 seeds.
Results. The evaluation results presented in Table 2 and Figure 9 further validate the efficacy of our proposed improvements in enhancing DA for more sample-efficient visual RL. Different weather conditions not only provide a rich set of realistic observations but also introduce diverse dynamics within the environments. In general, Rand PR outperforms the original DrQ-V2 with PadCrop as DA, indicating that preserving boundary information can effectively reduce the hardness of DA and thus improves sample efficiency in realistic environments. Furthermore, CycAug outperforms the previous SOTA algorithm by substantial margins 23.8% in final performance and 43.77 7% in low data regime.
5.3 Ablation Study and Detailed Analysis.
Training Stability. RL training highly sensitive to data stability [67]; hence, when fusing various DA operations to enhance diversity, it is imperative to minimize frequent distribution shifts to ensure stability. To quantitatively assess training stability, we measure the Q Variance and Q-Target Variance during the training process, adhering to the experimental methodology outlined by [68]. The results presented in Figure 10 demonstrate that CyeAug can optimally preserve training stability in comparison to other fusion approaches.
Expanded DA Quantities and Types. We primarily demonstrate the performance of CycAug when fusing PadCrop and Rand PR as components. However, as multi-type DA fusion scheme, CycAug

---

MultsTyge yype
Cycling Augmestation
Q-Target Variance
Composing Augnentation
Variance
Sumpling Auquaeniation
Mixing Augnscttiation
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Frame
Frame
1+6
Figure 10: Comparison of Q Variance and Q-Target Variance during the training process with CycAug and other multi-type DA fusion schemes. Variance consistently assessed through forward passes, employing random data augmentation on identical observations. CycAug demonstrates notably superior data stability, which crucial during the training process of visual RL.
can integrate a wider variety of DA operations in greater quantities, provided these DA operations are individually effective. We illustrate the scalability of CycAug in Figure 11. Three-component CyeAug utilizing PC, PR, and CS attains superior sample efficiency versus the componnt CycAug with PC and PR, exhibiting the capacity for additional expansion of CycAug.
Quadruped Run
Quadruped Run
PudCrrp
Compeaes:
PM
Padifeeiion
PC+PR
Eyaadi Natumn
CrupShtt
PC
Trunslae (Tc)
33.8
rC+
100
110
200 Episode Return After 2M Stopo
300
Prane
Frans
Figure 11: Expanded DA components integrated by CycAug. (Left) More Types. Employing CycAug fuse the commonly employed DA approach of PadCrop (PC) with other DA operations reveals varied performance outcomes. (Right) More Quantities. The effect of further integrating the original CycAug (PC+PR) with the third DA operation depends on the effectiveness of using that DA alone. PC+PR+CS surpasses PC+PR, highlighting CycAug's potential for further enhancement.
Conclusion
In conclusion, this work investigates the critical DA attributes for improving sample efficiency in visual RL and proposes two methods to enhance its effectiveness. We find that both spatial diversity and slight hardness crucial for individual DA operations, and introduce Random PadResize (Rand PR) as a new transformation that satisfies these attributes. We also propose a multi-type DA fusion scheme, referred to Cycling Augmentation (CycAug), which employs cyclic application of different DA operations to enhance type diversity while ensuring data distribution consistency for stability. The evaluations sample efficiency of CycAug with Rand PR as a key component, surpassing previous SOTA methods. Furthermore, this work demonstrates the potential of enhancing sample-efficient visual RL through more effective DA operations, without modifying the underlying RL algorithm or network architecture. Therefore, we advocate for greater emphasis on data-centric methods in the visual RL community.
the DeepMind Control suite and CARLA demonstrate the superior
Linitations. We investigate the essential attributes of DA for sample-efficient visual RL. However, further research is needed to understand how these attributes impact visual RL training fundamentally.
Acknowledgements This work is supported by STI 2030-Major Projects (No. 20212D0201405), We thank Yunjie Su, Zixuan Liu, and Zexin Li for their valuable suggestions and collaboration. We sincerely appreciate the time and effort invested by the anonymous reviewers evaluating our work, and are grateful for their valuable and insightful feedback.
10

---

References
[1] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in nearal information processing systems, 31, 2018.
[2] Denis Yarats, Amy Zhang. Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sam- ple efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
2021.
[3] Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual reinforcement learning. arXiv preprint arXhv:2210.04561, 2022.
[4] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 1000044.3344 13649, 2020.
[5] Denis Yarats. Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint wrXivv2107.09644 2021.
[6] Sicong Liu, Xi Sheryl Zhang, Yushuo Li, Yifan Zhang, and Jian Cheng. On the data- aata-efficiency with contrastive image transformation in reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[7] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint 2rXiv:2007.05929, 2020.
[8] Xiang Li, Jinghuan Shang. Srijan Das, and Michael Ryoo Does self-supervised learning really improve reinforcement learning from pixels? Advances in Neural Information Processing Systems, 35:30865 30881, 2022.
[9] Nicklas Hansen, Zhecheng Yuan, Yanjic Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning- weating--rro-see from-scratch baseline. arXiv preprint arXiv:2212.05749, 2022.
[10] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeal, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 19:19984-19895, 2020.
[11] Haoning Xi, Yi Zhang, and Yi Zhang. Delection of safety features of drivers based image processing. In 18th COTA International Conference of Transportation Professionals, pages 2098-2109, American Society of Civil Engineers Reston, VA, 2018.
[12] Yangru Huang. Peixi Peng, Yifan Zhao, Guangyao Chen, and Yonghong Tian. Spectrum random masking for generalization in image-based reinforcement learning. Advances in Neural Information Processing Systems, 35:20393-20M66, 2022.
[13] Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Aug- menting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34:5276-5289, 2021.
[14] Tao Yu. Zhizheng Zhang, Cuiling Lan, Yan Lu, and Zhibo Chen. Mask-based latent reconstruction for reinforcement learning. Advances in Neural Information Processing Systems, 15:25117-25111 2022.
[15] Jiameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi- view information bottleneck. In International Conference on Machine Learning, pages 6074-61044 PMLR, 2022.
[16] Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference Machine Learning, pages 6639-5650. PMLR, 2020.
[17] Chenyu Sun, Hangwei Qian, and Chunyan Miao. Celf: A contrsstiveecuurriiiyy.veeee learning framework for sample-efficiett reinforcement learning. arXiv preprint arXiv:2205.0094: 2022.
[18] Lin Li and Michael W. Spratling. Data augmentation alone can improve adversarial training. In The Eleventh International Conference on Learning Representations, 2023.
[19] Haotao Wang, Chaowei Xiao, Jean Kossaiti, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. Advances in neural information processing systems, 34:237-2500 2021.

---

[20] Raphael Gontijo-Lopss, Sylvia Smullin, Ekin Dogus Cubuk, and Ethan Dyer. Tradeoff's in data augmenta- tion: An empirical study. In International Conference Learning Representations, 2021.
[21] Jachyung Kim, Dongyeop Kang, Sungsoo Ahn, and Jinwoo Shin. What makes better augmentation strategies? augment difficult but not too different. International Conference on Learning Representations, 2021.
[22] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint prXiv:1801.000 2018.
[23] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1 1-16, PMLR, 2017.
[24] Kuang-Huei Lee, lan Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems. 33:11890-11901. 2020.
[25] Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, Tie-Yan Van Liu, and Houqiang Masked contrastive representation learning for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 0007/00773338 2018.
(27] Thanh Nguyen. Tung M Luu, Thang Vu, and Chang D Yoo. Sample-cffiiien reinforcement learning representation learning with curiosity contrastive forward dynamics model. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3471-3477. IEEE, 2021.
[28] Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. In Advances in Neural Information Processing Systems, 2020.
[29] Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vr1:: A data-driven framework for visual deep reinforcement learning. arXiv preprint arXiv:2202 10324, 2022.
[30] Rutav M Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement leaminn. In International Conference on Machine Learning, pages 9465-9476. PMIRR, 2021.
[31] Tete Xiao, Lija Radosavovic, Trevor Darrell, and Jitendra Malik, Masked visual pre- pre-training for motor control. arXiv preprint arXiv:2203.06/73 2022.
[32] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference Machine Learning. pages 17359-1733 17371. PMLR, 2022.
[33] Suraj Nair. Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
[34] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 1 11920-11931. PMLR, 2021.
[35] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee. and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pages 1332-1344. PMLR, 2023.
[36] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arxivv 2019.
[37] Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-fres model-based reinforcement learning with prototypical representations. In International Conference Machine Learning, pages 4956-4975. PMLR, 2022.
[38] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02/93, 2020.
[39] Danijar Hafner, Jurgis Pasukonis. Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint 001402301..4104.00 2023.
12

---

[40] Haoning Xi, Liu He. Yi Zhang. and Zhen Wang. Differentiable road pricing for environment--riented electric vehicle and gasoline vehicle in the bi-objective transportation network. Transportation Lettery, 14(6):60-677, 2022.
[41] Philipp Wu, Alejandro Escontrela. Danijar Hafnner, Pieter Abbeel, and Ken Goldberg, Daydreamer: World models for physical robot learning. In Conference on Robot Learning, pages 2226-2240. PMIRR, 2023.
[42] Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Paz. mixup: Beyond empirical risk minimization, arXiv preprint 17
2017.
[43] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[44] Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation strategies from data. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113-123, 2019.
[45] Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of of-the-arr art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 774-782, 2021.
[46] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 112.:1912.02771, 2019.
[47] Huan Wang, Suhas Lohit, Michael N Jones, and Yun Fu. What makes "good" data augmentation in knowledge distillation-a statistical perspective. Advances in Neural Information Processing Systems. 15:13456-1369, 2022.
[48] Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang. Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16783-16792, 2022.
[49] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 34:29935-29948, 2021.
[50] Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 379-389, 2023.
[51] Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for nlp. arXiv preprint arXiv:2105.0300 03005 2021.
[52] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Yan Zhu, and Song Han. Differentiable augmentation for data- efficient gan training. Advances in Neural Information Processing Systems, 33:7559- -570, 2020.
[53] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yl Yi Lin, Ekin D Cubuk, Quac V Le, and Barret Zoph. Simple copy-paste strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2918-2928, 2021.
[54] Alexander Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learning to compose domain-specife, transformations for data augmentation. Advances neural information processing systems, 30, 2017.
[55] Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[56] Tom Ching LingChen, Ava Khonsari, Amirreza Lashkari, Mina Rafi Nazari, Jaspreet Singh Sambee, and Mario A Nascimento. Uniformaugment: A search-free probabilistic data augmentation approach. arXiv preprint arXiv:2003. 14348, 2020.
[57] Teppei Suzuki. Teachaugment: Data augmentation optimization using teacher knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10904-10914, 2022.
[58] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304 222200222220......22 2021.
13

---

[59] Edoardo Cetin, Philip Ball. Stephen Roberts, and Oya Celiktutan. Stabilizing off off-policy deep reinforce- ment learning from pixels. In International Conference Machine Learning, pages 2784-2810. PMLR. 2022.
[60] Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pages (6828-1684)) PMLR, 2022.
[61] Byungchan Ko Jungseul Ok. Efficient scheduling of data augmentation for deep reinforcement learning. arXiv preprint arXiv:2206.00516 2022.
[62] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Fei-Fei, Yuke Zhu, and Anima Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. arXiv preprint arXiv:2106.06677 2021.
[63] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforeement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[64] Jiaxian Guo, Mingming Gong, and Dacheng Tao. A relational intervention approach for unsupervised dynamics generalization in model-basee reinforcement learning. In International Conference Learning Representations, 2022.
[65] Ziheng Wang and Justin Sirignano. Global convergence of the ode limit for online actor-critic algorithms in reinforcement Icarning. arXiv preprint arXiv:2108.08653 2021.
[66] Amy Zhang. Rowan McAllister, Roberto Calandra. Yarin Gal, and Sergey Levine. Learning invariant representations reinforcement learning without reconstruction. arXiv preprint arXiv:2006.1074 2020.
[67] Evgenii Nikishin, Pavel Izmailov, Ben Athiwaratkun, Dmitrii Podoprikhin, Timur Garipov, Pavel Shvechikov, Dmitry Vetrov, and Andrew Gordon Wilson. Improving stability in deep reinforcement learning with weight averaging. In Uncertainty in artificial intelligence workshop on uncertainty Deep learning, 2018
[68] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in neural information processing systems, 34:3680-3693, 2021.
14

---

Supplementary Material for
Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning
In this material, we provide additional information on previous related work and background knowl- edge, as well as detailed experimental setup and results. Appendix A provides an extensive review of relevant literature, offering comprehensive background that helps contextualize the significance and positioning of this paper. In Appendix B, we present a comprehensive analysis of decoupling and accurately controlling the hardness and diversity of DA, forming the basis for our experimental framework and further investigations. In Appendix C and Appendix D, we present comprehensive information regarding the experimental setup, hyperparameters, and detailed results for the rethinking and evaluation parts, respectively.
A Extended Related Work
A.1 What Characterizes Good DA
DA has become a ubiquitous technique in the whole deep learning community [42, 46, 43). After decades of rapid development, there has been recently increasing attention focused on reevaluating which properties contribute to the effectiveness of DA [18-21, 47]. Rethinking "what makes good DA" not only enhances our understanding of how DA works but also guides the development of more effective techniques. [20] first introduced two easy-to-compute metrics, affinity (the inverse of hardness) and diversity, to decompose the attributes of DA and revealed that augmentation performance is determined by both affinity and diversity. Following this analytical framework, [18] demonstrated that diversity has the capacity to improve both accuracy and robustness. Contrarily, hardness can enhance robustness at the expense of accuracy, up to a certain threshold, beyond which both accuracy and robustness may be compromised. This paper conducts a comprehensive analysis to investigate the specific requisites of hardness and diversity in DA for sample-efficient visual RL. unveils fundamental distinctions compared to other domains: (1) Visual RI. training is highly sensitive improvements in hardness, which is distinct from the fact that a certain level of hardness is necessary to improve robustness in supervised learning scenarios [19, 18]. (2) Increasing strength diversity and type diversity without considering the data-sensitive nature of RL training may not effectively enhance DA's performance in visual RL.
Apart from examining the essential attributes of good DA methods in terms of hardness and diversity. there have been studies dedicated to identifying additional indicators that can assess the effectiveness of DA in specific scenarios. In natural language processing (NLP) tasks, DND [21] recommends that good DA operations should generate diverse and challenging samples to provide informative training signals, while preserving the semantics of the original samples. In knowledge distillation (KD) scenarios, [47] conducted a statistical investigation and found that an effective DA scheme should minimize the cross-entropy variance between the teacher and student models. Exploring the RL-tailored metrics to assess the effectiveness of DA in sample-efficient visual RL represents a promising direction for future research.
A.2 DA Design in Sample-Efticient Visual RL
There are two primary works that compare the impact of different DA operations on the sample efficiency of visual RL. DrQ [4] evaluates the effecttivneess of 6 popular augmentation techniques, including random shifts, vertical and horizontal flips, random rotations, cutouts and intensity jittering. RAD [10] investigates and ablates crop, translate, window, flip, rotate, grayscale, cutout, cutout- color, random convolution, and color jitter augmentations. Following the classification criteria for DA operations in survey [3], the conclusions derived from the analysis of DrQ and RAD can be summarized as follows: (1) The photometric transformations, such as color jitter, and the random erasing operations, such as cutout, are generally found to be ineffective for sample-efficient visual RL. (2) Geometric transformations, represented by crop and shift operations, exhibit the strongest efficacy in significantly improving the sample efficiency. Despite the recognized effectiveness of
15

---

geometric transformations, there remains a scarcity of research dedicated to exploring the fine-tuning of operation details and parameters DA. This study aims to derive actionable guidelines for the DA design by investigating the essential attributes of DA required for achieving sample-efficien visual RL. Based on our thorough analysis, we propose more effective DA operation called Random PadResize (Rand PR). Compared with the shift operation in DrQ [4] and the crop operation in RAD [10], Rand PR exhibits higher spatial diversity by introducing randomness not only in positioning/transllaae also in the scaling factor. Meanwhile, Rand PR effectively mitigates the DA hardness by preserving edge information observations.
but
Additionally, [8] provides evidence supporting the assertion that employing DA alone achieves higher sample efficiency compared to jointly learning additional self supervised losses. To substantiate this claim, the authors conduct ablation experiments to analyze the distinct effects brought about by various DA. The observed phenomena in the DM Control tasks can be explained by the conclusions drawn from our study: In the case of the crop operation, where large-sized original rendered observations are randomly cropped to x 84 images as augmented data, it is evident that the augmentation effect noticeably superior when the original size is 100 100 opposed to 88 x 88. This can be attributed to the fact that in DM Control tasks, where edge information largely task-irrelevant, a slightly larger original size can promote higher spatial diversity, thereby improving the DA effectiveness.
A.3 Multi-Type DA Fusion Schemes
Beyond individual transformations, recent research has shifted towards combining different DA operations to achieve more diverse and powerful augmentation, which is termed multi-type DA fusion. The common fusion schemes can be categorized into three types: (1) Composing- -aased fusion is fundamental fusion scheme that combines different types of DA operations sequentially, either in a fixed or random order [54, 55]. (2) Sampling-bar based fusion applies a single augmentation each image, with the DA operation and strength (uniformly) sampled from the corresponding spaces [56]. TrivialAugment [45] serves as a strong representative example of this scheme. (3) Mixing-based fusion, exemplified by AugMix [46], randomly combines diverse augmented data generated by different operations to produce the final augmented data [19, 48]. Figure 12 illustrates the visual representation of the flow process for these three fusion schemes.
Composing- based Fusion
Sampling- based Fusion
Mixing- Fusion
Rotate
Translate
Rotate
Translate
Rotate
Translate
xa
Sample random mixing ratio
Sample one individual operation and apply it
Figure 12: Illustration of composing-based sampling-base and mixing-based fusion schemes.
Automatic DA Combined with Multi-Type DA Fusion The aforementioned fusion methods can be further improved by combined with automatic DA techniques, such as adversarial training (AugMax [19]) and hyper-parameter search (AutoAugment [44]). However, automatic DA comes with additional computational costs, which can be unfriendly to RL training that is already computationally expensive. Moreover, recent studies indicate that increased type diversity and randomness appear to be crucial attributes for superior DA performance [45, 18, 55]. Therefore, this paper focuses on the random fusion of multi-type DA.
16

---

B Hardness and Diversity in Visual RL
In this section, we provide supplementary details on the decomposition of DA into hardness and diversity components. We demonstrate the existing linear correlation between the hardness level and transformation strength of individual DA operations; and introduce the specific methodologies for manipulating the strength diversity and spatial diversity of DA.
B.1 The Relationship between Hardness and Strength
Hardness initially introduced metric to quantify the data distribution shift caused by DA [20]] defined as the ratio of performance drop on augmented test data for a model trained on clean data. In contrast supervised learning. where DA is employed to enhance the robustness and generalization of models, in many visual RL scenarios, it is simply not possible to obtain effective policies without DA. Hence, except for exceptionally simple tasks like Cartpole Balance, the initial definition is not applicable for measuring the hardness level of DA operations.
Cartpole Balance (TranslateX)
Caripole Balance (TranslateY)
Cartpole Balance (Rolate)
Cemvaaton
575
Sandasss
Santoss
Stewngth
Stemogtth
Strengtth
Figure 13: Relationship between Hardness and Strength of DA
It is necessary to identify controllable variable that can approximate the control over the hardness level of DA operations. Inspired by previous research [18]. demonstrating the strength a transfor- mation affects its hardness, aim investigate the relationship between the strength of individual DA operations and their corresponding hardness. We perform validation experiments in the Cartpole Balance task, where the visual RL algorithm can achieve satisfactory performance even in the absence of DA. As depicted in Figure 13, we observe a strong linear correlation between the strength and hardness of different DA operations. Therefore, is reasonable manipulate the strength of DA operations in order to control their hardness level and thereby investigate the impact of hardness on the effectiveness of DA for sample-efficient visual RL.
B.2 Strength Diversity and Spatial Diversity
Diversity of the strength can achieved by defining distinct ranges of hardness, as illustrated in Figure 14 (left). During the training process, each image is augmented with randomly sampled strength value from the specified range. Similarly, spatial diversity is manipulated by defining a set
Allowed Strength Range
Allowed Translate Directions
0
Strength D=1
Spatial D=1
9,10, 11
Strength D=
Spatial Dm
8, 90, 11, 12
Strength D=3
Spatial D=4
7,8, 9, 9,10 12, 13
Strength D=4
Spatial D=8
Constant Average Strength
Figure 14: Illustration of the manipulation of strength diversity and spatial diversity DA.
17

---

of allowable spatial operations, with the size of the set determining the level of spatial diversity. As depicted in Figure 14 (right), spatial diversity of indicates that only the Translate operation is allowed to move the observation in a single pre-determined direction.
C Detailed Experiment Results of the Rethinking Part
In this section, we present a more detailed description of the experimental setup implemented in Section 3.2 and provide comprehensive results from comparative experiments.
C.1 Customized DA with Controllable Hardness and Diversity
To investigate the impact of DA attributes the sample efficiency of visual RL, it is necessary to rigorously control variables, ensuring that while one attribute changed, the other attributes remain constant. For individual DA operations with predetermined type diversity, it is important accurately control their level hardness and spatial diversity. Inspired by the experimental design in [18], we propose PadResize-Hi CropShift-H) and Translate-HD which enable precise control over the hardness level through strength and the spatial diversity level through degree of freedom.
CropShift- HD operates by randomly selecting a region in the image and subsequently shifting to random position within the input space. The shape of the cropped region can be square or rectangular. The strength of CropShift-HD is determined by the parameter H, which represents the total number of cropped rows and columns. For instance, when setting the strength to 8, CropShift-HD removes lines from the left, right, top, and bottom borders of the image, where the total number of removed lines is equal to 8. This strength parameter allows for precise control over the hardness level and strength diversity of the operation. Furthermore, during initialization, we pre-sample D sets of operation parameters for crop and shift to control the spatial diversity of CropShift SrpphifftHH.... Subsequently, during the actual operations, the parameters are randomly selected from the pre-initialln initialized D sets.
PadResize-HDP starts by adding specified number of pixels in each direction of the image and then resizes the image back to its original size. Its hardness level is determined by the cumulative number of pixels added in each direction (i.e., top, bottom, left, and right). Given the hardness level, we pae-sample D sets of pad parameters for each direction to control the spatial diversity of PadResize HD. Note that when employing Rand PR, proposed in Section 4, which offers unrestricted spatial diversity, there no requirement to pre-sample D sets of pad parameters. Instead, the parameters for each operation are randomly sampled from the entire sets of possibilities each time the operation is executed.
Translate-HD involves shifting the observations by certain number of pixels in both the horizontal and vertical directions. The strength of the Translate-il is defined as the total number of pixels moved in both directions, while its spatial diversity is determined by the allowed directions of movement, Specifically, all the allowed directions include top f. bottom 4. left <-, right -t top-left K, top-right 7. bottom-left /, and bottom-right ' providing eight degrees of freedom. We randomly select D directions from these options to control the spatial diversity.
C.2 Experimental Setup of the Rethinking Part
We conduct the investigation in the DeepMind Control (DMC) suite [22] and based on the previously superior DrQ-V2 algorithms [5]. We maintain all hyperparameters from DrQ-V DrQ-V2 [5] unchanged and solely manipulate its DA operations. For detailed network architecture and hyperparameter configuration, please refer to Appendix D.I.
C.3 Additional Results of Hardness Investigation
We present the detailed training curves for different hardness levels of DA operations and the corresponding trend of final performance variation with respect to hardness levels in Figure 15 for the Cup Catch task, Figure 16 for the Finger Spin task, and Figure 17 for the Cheetah Run task.
18

---

Cup Catch (CropSaift-Hl) Strength D-2, Spatial D-1)
Cup Catch (CrogSait-HDD Strangth D-2, Spatial D-2)
Dladdoa Land
Catch (CropSaltt-HD. Strength
Cap (CropShit-HD) Strength D=2, Spattal D=
Hart
Cup Catch (PudResize-HD HD, Strength D=3, Spatial D=1)
Cup Catch (PadResize-Hi) HD, Strength D=3, Spatial D=2)
10
Thedoooc Leve)
Bacloss
(PadResize-HD, Strength Spatial
Cup Cabch (PadResiie-HD. Strength D=3, Spatial D=
Fritss
Horkras
Cup Catch (Translate-HID HDD, Strength D=2, Spatial D=1)
Cup Catch (Translate-HD Strength D=2, Spatial D=2)
Baedoos
Cup (Uranslate-HD. Strength Spatial
Cup Catch (Thamelate-ID. Strength Dm2, Spatial D=6)
Tross
Augmentation Hardness/Strength
Average Type
Hardnesse1 Hardnesss2
Hardnesse3
Top-5
Hardnesse4
Hardnesss5
Interquartile Mean
Figure 15: Detailed comparison training sample efficiency and final performance across different hardness levels using three individual DA operations the Cup Catch task.
19

---

Finger Spin (CropShit-HD. Strength D- Spatial 1)
Finger Spin (CropShift-Hl HDD, Streagth D=2, Spatial D-2)
Daadoaa Land.
Drase
Hardoaa Land
(CropShitt-HD,
Spatial
(CropShiit-HD) Streagth
200
Draze
Drazs
Finger Spin (PadBosize-HD Strength D=3, Spatial D=1)
Finger Spin (PudResizz--HD. Strungth D=3. Spatial D
bd
Baedooo Leee)
Booce
Bacloss
Finger (PadResizs-HD. Strength Spatial D=4)
Finger Spin (PadResize-HD,
Strength Spatial
20
Hacdoo Land
Finger Spin. (Tramslate-lie HDD, Strength D=2, Spatial D=1)
Finger Spin (Translate-Hle HD. Strength D=2, Spatial D=2)
Finger Spin. (Trusslate-HD.. Sizenyth D=2, Spatial D=
Finger Spin (Trandate-HDD Stzength Spetial
100
Backood Leed
Brocee
Augmentation Hardness/Strength
Average Type
Hardnesse1
Hardness=2
Hardnesse3
Moan Top-5
Hardnesse4
Hardnese:5
Interquartile Mean
Figure 16: Detailed comparison training sample efficiency and final performance across different hardness levels using three individual DA operations the Finger Spin task.
20

---

Cheetah Run (CrogShiftHHD. Strength D-2, Spatial D-1)
Cheetah Run (CropSaint-HD) H-DD Strength D-2, Spatial D-2)
250
10
Dladdoa Land
Praze
Hardoaa Land
Cheetah (CropSaint-HD. Strength
Cheetah Run (CropSairt-HiD
Spatial
200
Poaza
Chestah Run (PadResize-FFt HDD, Strongth D=3, Spatial D-1)
Chestah Run (PadRosize-FED Strength D=3, Spatial D=2)
Thedooo Leed
Doocs
Bacloss
Cheatah (RidResibe-HD, Strength Spatial
Chestah (PadResizn-HD. Strongth Spatial
Doucs
Chestah: Rua (Translate-IDD, Strength D=2, Spatial D=1)
Cheetah Run (Translate-PP HDD, Strength D=2, Spatial D=
Bacdoos
(Trunslate-HD, Strungth Spetinl
Cheetak (Trenslate-HD Strungih Spatial
Broee
Augmentation Hardness/Strength
Average Type
Hardnesse1
Hardness=2
Hardnesse3
Moan Top-5
Hardnesse4
Hardness:5
Interquartile Mean
Figure 17: Detailed comparison training sample efficiency and final performance across different hardness levels using three individual DA operations the Cheetah Run task.
21

---

C.4 Additional Results of Strength Diversity Investigation
We present additional comparative results across varying strength diversity in Figure for the Cup Catch, Finger Spin and Cheetah Run tasks.
Catch (CoopStan-HBD Haclossu=2,
Catch (PudResizu-HD) HD, Hardinuu
Cup Cutchh (TrmnslsteeD Horthmwasm2, Spatial De4))
Dup Cafcch (Rotate<HD, Herdnewam),
Spatial D-1)
Pinger Spin (CropSiath-IID, Hardneesm2, Spettal De4)
Pinger Spin IPadleestze-EDD Hardnessm2, Spottal De4))
Prene
Soough Dxertt
Stregt (irendy
Pinger Sptn [Iransiate HID, Harddooss-4 Spatta) D-8)
Pinger Spaz (Rotato-11.. IID,
Hardness- Spattal D-
Sruntty
Stregn
(CropShift HD, Hardness-2,
HED, Hardness-2. Spatial D-2)
Chestal Hardbassed, Spatial
Choctah (BotateHDD Hardness=2 Spatial
DA Strength Diversity
Average Type
Strength D=1 .... Strengtt D=2 .... Strength D-3
Mean Top-5 Mean
Strength Dof Strength De5
Interquarttle Mean (IQN))
Figure 18: Detailed comparison of the sample efficiency across different strength diversity levels.
22

---

C.5 Additional Results of Spatial Diversity Investigation
Detailed training curves depicting the performance for various levels of spatial diversity in DA operations, along with the corresponding trend of final performance with increasing spatial diversity, are presented Figure for the Cup Catch task, Figure 20 for the Finger Spin task, and Figure 21
for the Cheetah Run task.
Calds (Crupölik Barrinssww Stoonigh Dw2))
Cop (hadilaciow-ll) Dardnnse-),, Strwngth D--3
Cup (Tuuslate Hardneasw), Strongth
MW
Catch (CrySSull Strdhesss22
Strougtt
Strengt ITrassiate Huchresse,, Stough D=2)
Coild
Calth (CroyShihiB HDL
Strditess Stroagth
(Traniatedi Hardhessm3, D=2)
Catch (CnpShit--EE) Startonee--,, Sirwagth
Catch Hardacszed,
Cetch (Tronlsteein) Harddweadd, D-
Cop Eatch (CropShit--11) Startinnes-S, Strongth
Cup Catoh (RoRes
Bardnosss6G
Catch [Imoniateeta Haarthees,, D-2)
Corld
DA Spatial Diversity Spotial D=1 Spatial Do2
Average Type
-M Top-5 Mean
Spatial Spatial
Interquartile Mean (IQM)
Figure 19: Detailed comparison of training sample efficiency and final performance across different spatial diversity levels on the Cup Catch tasks.
23

---

Flnger Steoogth
Fager Spin
2D, Bar docaue Sirength Dw5)
Finger
Strength
(Cropöhitt-II) [terdneee-12 Strongth
Hardnesse2
Finger (Thnndalee(II) Hlrrtosee-2 22 Strength D-2)
Spaid Dovic
Spatia Dessit
Spetei Dirrec
Pinger Spin [Cropthhtt-II)) [tertneee-1. Strongth D-12)
Fager Spin (PulRexino HD, Hardnrnoe Strongth Dw31
Fugger Spin (Thussltte HDD Hurdnza- Steoogth D=2
Spaia Druc
Spata Lecuty
Spraii Dessi
Flager (Crup6tult Blrinessed, Steoogth Dw2)
Pinger Spin (Padilacios-11)) Hiartnaser Strength D-31
Finger (Tiuzslate
Strength
(Cropéhif BardnesseG Steoogth
(Transate Hardness-G, D=2
Quatd Decii
Spatial Diversity
Average Type
Sporial
Sparial
-M Top
Interquartile (I0M)
Figure 20: Detailed comparison of training sample efficiency and final performance across different spatial diversity levels on the Finger Spin tasks.
24

---

Chretiah Bun CioySSull HD, Hanhasssw. Stooapth Dw2)
Chrebah Ren (Padileebw--ll, Linntraen-1, D-33
Clasriah Fun (Ii atualatte HD, Hundscool, Strospis Dw2
(Cuppttll Handonsw2, Stroogth
(Padilestan-II Lantiaase-2, Strongth D-01
Ran Hardsesse,, Stroapth
SediaDrinly
Statil
Chrctiah Bus (CoypStul HD, Hanhanssm,, Stroagth Dw2)
Cheetah Rua (PulRtssio HD, Hunlsesse,, Streugth De3t
Cheetah Rin (Tranilatecin,, [anttaee-)), Straagh D-
Strwagth
Stroogth De3)
Streagth
Gatal
SutaD
Roda Conaty
Cheetab Hoe (Crop Stuth LDD Hantnea- Stropta D-12)
Cheetah Rua (PolResioo 8D, Banloesses, Streugth De3t
Clostall Ran (Trasalate HD, Hardsewon, Stroagh Dw2)
qutuauenty
DA Spatial Diversity Spatial D-1 .... Spatial D=2
Average Type
Mean -M Top-5 Meea
Spatiol D=3
Spatial De
Interguartile Moan (IQ(A)
Figure 21: Detailed comparison of training sample efficiency and final performance across different spatial diversity levels on the Cheetah Run tasks.
25

---

C.6 Additional Results of Type Diversity Investigation
We present additional results in Table C.6 showcase the detailed performance of employing multi-type DA operations with three different fusion schemes.
Table 3: Detailed results of employing three different fusion methods in the Quadruped Run task, including the mean and standard deviation of performance across 10 random seeds.
Composing-based
PC+PR
PC+CS
PC+Tr
PC+Ro
PC+Co
PR+CS
PR+Tr
Fusion Scheme
461.7
354.0
350.7
468.2
309.2
PR-Ro
PR4Co
CS+Tr
CS+Ro
CS+Co
Tr4Ro
TN+Co
RosCo
409.8 = 173
369.7 + 140
396.2 = 168
463.1 157
331.5 + 146
382.5 4 215
302.6 +: 161
263.6 = 173
Sampling-basse
PC+PR
PC+CS
PC+Tr
PC+Ro
PC+Co
PR+CS
PR+Tr
Fusion Scheme
525.7 +:
490.8 + 143
487.7 145
492.8 126
351.7 + 149
475.8 + 159
497.1 113
PR+Ro
PR+Co
CS+Tr
CS+Ro
CS+Co
Th+Ro
Tr+Co
Ro+Co
491.6 98
357.3 + 172
446.6 + 113
417.5 = 149
298.0 166
430.1 + 161
349.0 + 149
256.7 + 159
Mixing-based
PC+PR
PC+CS
PC+Tr
PC+Ro
PC+Co
PR+CS
PR+Tr
Fusion Scheme
406.1 + 164
391.8 156
374.2 = 215
413.4 # 198
334.4 + 173
411.1 + 97
388.4 + 213
PR+Co
CS+Tr
CS+Ro
CS+Co
TMRo
TrCoo
Ro+Co
401.3 : 177
331.7 + 165
366.3 + 173. 400.4 = 143 339.9 = 171
3113 131
312.2 161
288. + 112
D Detailed Experimental Setup of the Evaluation Part
In this section, we provide comprehensive implementation details of our experimental setup for evaluating the performance both DMC and CARLA environments. This includes the introduction of benchmarks, network architecture, and hyperparameters.
D.1 Setup of DeepMind Control Suite
Benchmarks. We conduct evaluations 12 challenging continuous control tasks, following the task setting of DrQ-V2 V2 [5]. These tasks cover different types of tasks and various challenging elements. A detailed description is presented in Table D.I.
Table Detailed descriptions of the 12 DMC tasks leveraged for evaluation.
Task
Traits
dim(S)
dim(A)
Acrobot Swingup
diff. balance, dense
Cartpole Swingup Sparse
swing, sparse run, dense
1
Cheetah Run
18
6
Finger Turn Easy
turn, sparse
6
2
Finger Turn Hard
turn, sparse move, dense
6
2
Hopper Hop
14
4
Quadruped Run
run, dense walk, dense manipulation, sparse reach, dense
56
12
Quadruped Walk
56
12
Reach Duplo
55
9
Reacher Easy
4
2
Reacher Hard
reach, dense
4
2
Walker Run
run, dense
18
6
To demonstrate the general applicability of our method, we keep all environment-specifie hyperparameters from DrQ-V2 [5] unchanged and solely replace the orig- inal DA operations with Rand PR and CycAug. The hyper-parameters of DA operations are also
Hyper-parameters.
26

---

presented in Table 5. While we aim to maintain consistent settings across all tasks, we observe that the cycling interval of CycAug needed to be adjusted for more challenging tasks. Consequently, we set the cycling interval to 2 x 10 steps for the Quadruped Walk, Quadruped Run and Hopper Hop tasks. CycAug, a naive attempt guided by the insights that effective multi-type DA fusion in sample-efficient visual RI. should consider the data-sensitive nature of RI,, demonstrates the effectiveness of reducing the frequency of DA operation switching for training stability, This aligns with our rethinking conclusions. Our attempt provides preliminary validation of the feasibility of employing multi-type DA operations sample-effiiinnt visual RL, surpassing the limitations of individual DA approaches. There is still significant potential for future research explore and refine more suitable fusion schemes.
Table 5: A default set of hyper-parameter used in DMC evaluation.
Algorithms Hyper-parameters
Replay buffer capacity
106
Action repeat
2
Seed frames
4000
Exploration steps
2000
n-step returns
3
Mini-batch size
256
Discount ?
0.99
Optimizer
Adam
Learning rate
10
Agent update frequency
2
Critic Q-function soft-update rate т
0.01
Features dim.
50
Repr. dim.
32 x 35 x 35
Hidden dim.
1024
Exploration stddev. clip
0.3
Exploration stddev. schedule
linear 0.1, 500000)
DA Operations plper-parameters
Strength range of Rand PR Cycling interval of CycAug
[0, 16)
10 steps
D.2 Setup of CARLA Simulator
Benchmarks. Our experimental setup in CARLA is adapted from the work of [66]. Specifically, we employ three cameras mounted on the roof of the vehicle, with each camera capturing 60-degree field of view, as illustrated in Figure 22. These cameras are utilized for both training the agent and evaluating the final performance of the different methods. The reward function defined as:
Ft- V Vego ühighway At impulse A |steer].
where Vego represents the velocity vector of the ego vehicle projected onto the unit vector ühighway that aligns with the highway direction. is multiplied by the time discretization At 0.05 to measure
3x66x84
3x
94
B4x252
DA
Network
Figure 22: Pipeline of DrQ-V2, Rand PR and CycAug in CARLA. The size of the original observa- tions is 84 x 252, and we first resize them to 3 x 84 x 84 before applying DA operations. After DA operations, we resize the observations back to their original size and then input them to the network.
27

---

the progress of the vehicle on the highway in meters. Collisions are quantified in terms of impulses, measured Newton- sewton-seconds. Additionally. steering penalty imposed, with steer [-1, 1]. The reward function incorporates weights A; 10-1 and A3 1.
To adapt the DrQ-V2 algorithms CARLA tasks, partition the original observations, which of size 84 x 252, into three square images of size 84 x 84, each corresponding to the content captured by different camera. These individual images are then subjected DA operations, as shown in Figure 22. We evaluate our proposed Rand PR and CycAug methods against the PadCrop operation originally used in DrQ- V2 in four different weather settings. These varied weather conditions not only offer diverse range of realistic observations but also introduce different dynamics within the environments, The Figure 23 illustrates the diverse and realistic observations obtained from the four weather settings we leveraged in our evaluation.
Default
SoftRainNoon
HardRainšunset
Figure 23: Illustration realistic observations captured in CARLA under various weather conditions.
slyper-parameters. We adopt various hyperparameter settings from DrQ-V2 in the DMC tasks, only modifying the dimensions of the intermediate layers in the network to accommodate the observation size CARLA. The complete hyperparameter settings are presented in Table 6.
Table 6: A default set of aaper-parameten used in CARLA evaluation.
Algorithms Hyper- slyper-parameters
Replay buffer capacity
105
Action repeat
4
Exploration steps
100
n--tepp returns
3
Mini-batch size
512
Discount ?
0.99
Optimizer
Adam
Learning rate
10-4
Agent update frequency
2
Critic Q-function soft-update rate T
0.01
Features dim. Repr. dim.
50
32 x 35 x 119
Hidden dim.
1024
Exploration stddev. clip
0.3
Exploration stddev. schedule
linear(1.0 0.1, 100000)
DA Operations Hyper-paramette
Strength range of Rand PR Cycling interval of CycAug
[0, 16)
2 x 1 steps
28


Reviews:
[{"review_id": "GzQKyKwpNe", "reviewer": "Reviewer_RHoY", "summary": "The paper presents a thorough empirical analysis of visual data augmentations and their effects on RL training. They benchmark various spatial augmentation on two axes of variation, spatial diversity, and hardness, measured by the amount of distortion created in the image. The authors perform a series of experiments to benchmark the effect of augmentations along these two axes and propose best practices for training visual RL policies. Additionally, the authors offer a new data augmentation named Random Pad Resize and empirically demonstrate its benefits. They also propose a multi-DA fusion scheme, named CycAug which boosts sample efficiency even further and prevents training instability. ", "questions": "I would be happy to hear the authors' thoughts on the points mentioned in the weakness section. ", "limitations": "I would encourage the authors to include a section in the main paper or supplementary about the potential societal impacts of their work. The section is currently which may or may not be against the Neurips policy.  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Works like these perform a systematic empirical analysis of a known technique to help bring the community on a common page about its usage are very useful. \n2. Proposal of the CycAug method for multi-DA fusion, is a simple, but clever idea, and in combination with RandPR shows state-of-the-art sample efficiency on two benchmarks.\n3. The paper is well-written and easy to follow.   \n4. The authors have provided code and an extensive discussion of their experimental setup in the supplementary material. \n\n", "weaknesses": "1. The combination of RandPR and CycAug shows improved performance over previous methods. I failed to find any analysis that isolates each part and analyzes its impact on existing methods. \n2. Given that the major contribution of this work is a thorough empirical analysis of existing data augmentation strategies, adding breadth to the experiments and including Embodied environments like Habitat or AI2THOR, or manipulation benchmarks like MetaWorld and studying the effect of augmentations would make this paper even better. Note that the absence doesn't make the work any less useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and thorough review. We added the suggested experiments to the `Response PDF`. In the following, we seek to address each of your concerns.\n\n----\n**W1:** *\"The combination of Rand PR and CycAug shows improved performance over previous methods. I failed to find any analysis that isolates each part and analyzes its impact on existing methods.\"*\n\n**A**: It is hoped that the following explanatory and supplementary experiments can more precisely delineate the individual effectiveness of Rand PR and CycAug.\n1. Rand PR is an **individual DA operation** like Translate, Rotate, etc. It can be considered as an improvement of the DA method used in the original DrQ-V2, providing ample spatial diversity while ensuring a low level of hardness. We have conducted extensive experiments on DMC and CARLA tasks to demonstrate that Rand PR achieves higher sample efficiency compared to the DA operation (PadCrop) used in DrQ-V2, as evidenced in `Figures 8 and 9` of our paper. We report here again a comparison of the average performance of Rand PR and PadCrop on DMC and CARLA after limited iteration steps. Note that the superior effectiveness of Rand PR as an individual DA operation is entirely independent of CycAug.\n\n    | Augmentation Type  | DMC @ 1500k Frames | CARLA @ 100k Steps |\n    |--------------------|--------------------|--------------------|\n    | CropShift (DrQ-V2) |$547.96$|$99.7$|\n    | Rand PR            |$588.75$|$110.8$|\n\n2. CycAug is a fusion scheme that aims to combine multiple different DA operations together to achieve higher type diversity while ensuring data stability during training. As a fusion scheme, the effectiveness of the individual DA operations incorporated in this fusion method determines the baseline performance of the method after fusion, as shown in `Figure 5` of our paper. This is why we selected PadCrop and Rand PR as components for CycAug in this paper, as these two DA operations demonstrated markedly superior individual effectiveness compared to other DA. However, this does not imply that CycAug must rely on Rand PR, nor that CycAug can only fuse two DA operations. In fact, any high-performing individual DA can be incorporated into the CycAug scheme.  As illustrated in the following table, combining PC with PR and CS using CycAug can achieve better performance than using them individually. However, since Tr has poor individual effectiveness, combining it with PC actually decreases the augmentation effect of PC.\n\n    | Augmentation Type | Return || Aug Component | Return |\n    |-|-|-|-|-|\n    | PadCrop (PC) | $570.9\\pm 121$ | | | |\n    | Rand PR |$602.3\\pm 96$|| PC+PR |$728.6 \\pm 64$|\n    | CropShift (CS) |$536.5\\pm 89$|| PC+CS |$586.6\\pm 83$|\n    | Translate (Tr) |$467.4\\pm 9$ || PC+Tr |$545.7\\pm 99$|\n    \n    Furthermore, we trialled a three-component CycAug utilizing PC, PR, and CS. As depicted in `Figure 3 (Left) of the Response PDF`, this configuration attained superior sample efficiency on Quadruped Run versus the dual-component CycAug (PC+PR) presented in our publication, exhibiting the capacity for additional expansion of CycAug.\n    \n    |Aug Component|Return|\n    |-|-|\n    |PC+PR | $728.6 \\pm 64$|\n    |PC+PR+CS|$783.9\\pm 46$|\n    |PC+PR+Tr|$677.0\\pm 16$|\n    |PC+PR+CS+Tr|$736.2\\pm 62$|\n\n---\n**W2:** *\"[...]adding breadth to the experiments and including Embodied environments like Habitat or AI2THOR, or manipulation benchmarks like MetaWorld and studying the effect of augmentations would make this paper even better.\"*\n\n**A**: Thank you for your timely suggestion. Adding more challenging experiments can indeed help us better demonstrate the efficacy of our proposed method. Beyond evaluations on the prevalent DMC benchmarks, we have implemented comprehensive experiments on the more practical autonomous driving task CARLA within the original paper. As a supplement, we have conducted evaluations on two challenging tasks, Lift and TwoArmPegInHole, in the table-top manipulation environments of Robosuite. We report the average episode return over 5 random seeds after training for 500k frames (with 2 action repeat) and present the complete 1M frames training curves in `Figure 1 of the Response PDF`. The experimental results demonstrate that CycAug achieves higher sample efficiency than the original DrQ-V2. \n\n| Augmentation Type | Lift | TwoArmPegInHole |\n|-------------------|------|-----------------|\n| DrQ-V2            |$253.9\\pm 125$|$295.2\\pm 56$|\n| CycAug            |$324.5\\pm 120$|$350.4\\pm 53$|\n\nIn addition to Robosuite, we also attempt to evaluate our method in the indoor visual navigation environments of Habitat and will make efforts to expand the Habitat experimental results in the future version of the paper."}, {"review_id": "ZiGAddKHx5", "reviewer": "Reviewer_EBtK", "summary": "This paper explores the conditions for achieving sample-efficient visual RL with data augmentation. Based on the findings, two guidelines are proposed: one emphasizes sufficient spatial diversity with minimal hardness, leading to the introduction of Rand PadResize. Additionally, the data-sensitive nature of RL training is considered when designing multi-type data augmentation fusion schemes in visual RL. Drawing inspiration from this guideline, a RL-tailored multi-type data augmentation fusion scheme CycAug is proposed.", "questions": "1、\tCan CycAug effectively handle more than three augmentation types? The paper does not provide specific information about the scalability of CycAug beyond three augmentation types. Further investigation or experimentation would be necessary to determine the feasibility and performance of CycAug when applied with a larger number of augmentation types.  \n\n2、\tDoes unlimited spatial diversity lead to an increase in the level of hardness?\n", "limitations": "What are the limitations of the proposed method? I know there is something about this in the paper. But it is not solid and broad.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper exhibits strong writing quality and clear communication of ideas. The content is well-structured and effectively presents the research findings. The authors' explanations and descriptions are concise, enabling easy comprehension of the key concepts and methodologies discussed. Overall, the paper demonstrates a high level of writing proficiency.\n\n\nThe paper presents a comprehensive analysis of effective augmentation techniques for visual reinforcement learning (RL). The authors delve into the topic with depth, examining the hardness and diversity of various augmentation methods and their impact on the performance of visual RL algorithms. The analysis is thorough, providing valuable insights into the benefits and limitations of different augmentation strategies in enhancing visual RL.\n", "weaknesses": "1、\tTo obtain specific values for \"Strength D\" and \"Spatial D,\" it is necessary to train a strategy on raw data before estimating diversity, as mentioned in line 137. Does this imply that training the strategy using the original, unmodified data precedes the selection of suitable augmentation techniques?  \n\n2、\tThe author asserts that CycAug promotes stability throughout the training process, which raises curiosity about the variance of Q values during training. It would be beneficial for the authors to provide further analysis of the stability results. Additionally, considering that SVEA is another method to enhance stability in the context of augmentation, it would be valuable to compare the advantages of both methods when utilizing the same augmentations. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and insightful feedback. We will address each of your comments and concerns below and also in our revised manuscript.\n\n----\n**W1**: *\"Does this imply that training the strategy using the original, unmodified data precedes the selection of suitable augmentation techniques?\"*\n\n**A**: No, pre-training a policy on clean data is not a prerequisite for selecting effective data augmentations in practice.\n\nAccording to the definition of Hardness (${\\rm{Hardness}}={\\mathcal{R}(\\pi, \\mathcal{M})}{\\big /}{\\mathcal{R}(\\pi, \\mathcal{M}^{\\rm{aug}})}$), calculating the Hardness of a certain DA operation requires first training a clean policy $\\pi$ on the unmodified environment $\\mathcal{M}$, and then testing the average return $\\mathcal{R}\\left(\\pi, \\mathcal{M}^{\\rm{aug}}\\right)$ of this clean policy on the augmented environment $\\mathcal{M}^{\\rm{aug}}$. However, for most challenging visual RL tasks, we cannot train an effective policy on the clean, unaugmented environment without using DA.  Therefore, when using DA on practical complex tasks, we cannot pre-train a clean policy and then calculate the DA's Hardness according to the definition.  \n\nConsistent with previous studies [1], we observe a **strong linear positive correlation** between the hardness level and strength of individual DA operations in visual RL (shown in `Figure 11 in Appendix B.1` of our paper).  Hence, we can control the **Hardness level and Strength Diversity** of DA by adjusting the average strength and allowed strength ranges of individual operations. In addition, **Spatial Diversity** can be manipulated by defining a set of allowable degrees of freedom spatial operations (for example, the allowed translation directions in the Translate operation).\n\n[1] Lin Li et al. Data augmentation alone can improve adversarial training. ICLR 2023\n\n---\n**W2-1:** *\"[...]the variance of Q values during training[...]\"*\n\n**A:** In `Figure 2 of the Response PDF`, we compared the variance curves of Q values and Q-Targets during training when applying CycAug versus other multi-type DA fusion methods. CycAug demonstrates notably superior data stability, which is crucial during the training process of visual RL.\n\n---\n**W2-2:** *\"[...]compare the advantages of both methods (CycAug and SVEA) when utilizing the same augmentations.\"*\n\n**A:** SVEA aims to enhance training stability when applying **strong data augmentation**, which is imperative for improving generalization in visual RL, but will heighten the difficulty of training. Its main approaches include (1) using only weak augmentation PadCrop (called random shift in its original paper) to compute Q-Target values, and (2) mixing data of observations augmented by strong augmentations with those augmented by only weak augmentations to reduce the difficulty of learning from augmented data. The core idea of these two approaches is to leverage weak augmentation to alleviate data instability introduced by strong augmentation.\n\nUnlike SVEA, the DA operations explored in our paper are weak augmentations from SVEA's perspective, which aim to improve sample efficiency during training rather than improve generalization by incorporating prior knowledge. The instability issue that CycAug aims to address occurs when using multiple different weak DAs simultaneously during training, which has the potential to further improve sample efficiency due to higher type diversity but also introduces additional data instability to the training process. Note that CycAug's approach for handling multi-type weak DA fusion can be further combined with SVEA's scheme for strong DA to jointly harness the utilities of both.\n\n---\n**Q1**: *\"Can CycAug effectively handle more than three augmentation types?\"*\n\n**A**: CycAug is a multi-type DA fusion scheme that can incorporate far more than three DA types, as long as these DA operations themselves have sufficient individual effectiveness. We conducted further experiments on the Quadruped Run task to demonstrate the effects of CycAug with three and four components, as shown in the table below. Training curves are illustrated in `Figure 3 (Left) of the Response PDF`. Three-component CycAug utilizing PC, PR, and CS attained superior sample efficiency versus the dual-component CycAug (PC+PR) presented in our paper, exhibiting the capacity for additional expansion of CycAug.\n\n|Aug Component|Return|\n|-|-|\n|PC+PR | $728.6 \\pm 64$ |\n|CropShift (CS)|$536.5\\pm 89$|\n|Translate (Tr)|$467.4\\pm 9$|\n|PC+PR+CS|$783.9\\pm 46$|\n|PC+PR+Tr|$677.0\\pm 16$|\n|PC+PR+CS+Tr|$736.2\\pm 62$|\n\n---\n**Q2**: *\"Does unlimited spatial diversity lead to an increase in the level of hardness?\"*\n\n**A**: Thanks for your very valuable question. In the original paper, we experimentally validated that the hardness level of individual DA operations is highly linearly correlated with the strength of their transformations. Thus we naturally inferred that the hardness remains unchanged as long as the augmentation strength is not altered. To validate this inference, we conduct a quantitative analysis of the DA's hardness on the CartPole Balance task. We manipulate the level of spatial diversity by specifying different spatial degrees of freedom with the same strength. We pre-trained a clean policy on the unaugmented CartPole Balance task, then tested the performance of this policy under different DAs and calculated the Hardness according to the definition. The results in the table below demonstrate that three different augmentation methods exhibit consistent hardness across varying levels of spatial diversity.\n\n| Spatial Diversity Level | 1 | 2 | 4 | 8 | w/o Limited |\n|-|-|-|-|-|-|\n| CropShift-HD|$1.50 \\pm 0.59$|$1.52 \\pm 0.66$|$1.56 \\pm 0.75$|$1.48 \\pm 0.42$|$1.53 \\pm 0.51$|\n| PadResize-HD|$1.32 \\pm 0.58$|$1.29 \\pm 0.72$|$1.36 \\pm 0.68$|$1.34 \\pm 0.51$|$1.33 \\pm 0.44$|\n| Translate-HD|$2.25 \\pm 0.71$|$2.27 \\pm 0.68$|$2.24 \\pm 0.79$|$2.30 \\pm 0.82$|$2.28 \\pm 0.63$|"}, {"review_id": "V4XrLCzjxS", "reviewer": "Reviewer_2eeL", "summary": "This paper explores the fundamental aspects of data augmentation (DA) in the context of visual reinforcement learning (RL) and introduces two methods, Random PadResize (Rand PR) and Cycling Augmentation (CycAug), to enhance its efficacy. Extensive experiments on the DeepMind Control suite and CARLA are conducted to showcase the superior performance achieved by the proposed methods.", "questions": "Please refer to the weakness.", "limitations": "Please refer to the weakness.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1, This paper is well-motivated. The paper tackles a pressing and significant issue of sample efficiency in visual reinforcement learning (RL), which poses a crucial obstacle for the real-world implementation of RL agents across diverse domains.  \n2, The paper presents a comprehensive and meticulous analysis of how the attributes of data augmentation (DA), including hardness, diversity, and fusion schemes, affect the sample efficiency of visual reinforcement learning (RL). Moreover, the paper highlights the unique requirements of DA for visual RL, distinguishing it from other domains such as supervised learning or adversarial training.  \n3, The writing is clear and easy to follow.   \n4, The paper showcases the effectiveness of the proposed methods on two demanding benchmarks for visual reinforcement learning (RL): the DeepMind Control suite and CARLA. These benchmarks provide challenging environments that allow for a thorough evaluation of the proposed methods.  ", "weaknesses": "1, This paper overlooks the comparison of the proposed methods with other existing data augmentation (DA) techniques explicitly designed for visual reinforcement learning (RL), such as Spectrum Random Masking [11] or PlayVirtual [12]. It would be intriguing to observe and evaluate the performance of the proposed methods in relation to these techniques in terms of sample efficiency, generalization, or diversity. Incorporating such a comparison would provide a more comprehensive understanding of the strengths and limitations of the proposed methods within the context of existing approaches for visual RL.  \n2, The paper lacks ablation studies or qualitative analysis that elucidate the individual contributions of each component or attribute of the proposed methods to their effectiveness. For instance, it would be valuable to compare Rand PR with PadCrop or Translate, examining factors such as hardness or spatial diversity. Additionally, evaluating CycAug against other fusion schemes in terms of type diversity or data stability would provide further insights. Including such ablation studies and qualitative analysis would enhance the understanding of the proposed methods and their specific strengths relative to alternative components or attributes.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and insightful feedback. We will address each of your comments and concerns below and also in our revised manuscript.\n\n----\n**W1**: *\"[...] observe and evaluate the performance of the proposed methods in relation to these techniques (such as SRM and PlayVirtual) in terms of sample efficiency, generalization, or diversity. [...]\"*\n\n**A**:  In visual RL tasks, using data augmentation is typically motivated by two goals: improving sample efficiency and enhancing generalization ability[1]. In this paper, we focus on investigating \"which attributes enable effective DA for achieving sample-efficient visual RL?\" and devise ways to harness the potential of DA in this regard. In addition to improving sample efficiency, there are also several works that aim to use DA to enhance the generalization ability of agents, such as SODA[2], SVEA[3] and SRM. These studies introduce stronger augmentations such as Overlay to improve generalization, which have been shown to be **inevitably detrimental to sample efficiency during training**[4]. The core contribution of SODA [2] and SVEA [3] lies in how to leverage weak augmentation to alleviate the negative impact of strong augmentation on the visual RL training process. Since the goal of this paper is to maximize the potential of DA for improving sample efficiency, we do not consider these strong augmentations that undermine sample efficiency as comparable targets. Note that our investigation and proposed methods are orthogonal to those DA methods aiming to improve the generalization ability of visual RL, and thus can be combined with them for further improvements.\n\nAdditionally, apart from methods like DrQ-V2 that only apply augmentation on the input without modifying other parts of the algorithm, there are also many works that combine DA with other self-supervised auxiliary tasks, such as PlayVirtual and SPR[5]. However, the latest work[6] demonstrates that adding explicit self-supervised learning tasks does not achieve higher sample efficiency compared to only applying DA as an implicit regularization. This further motivates our approach to focus on understanding and improving DA itself. This is also why we chose DrQ-V2 as our baseline and conduct in-depth research on DA based on it.\n\n[1] Ma G, Wang Z, Yuan Z, et al. A comprehensive survey of data augmentation in visual reinforcement learning. arXiv preprint arXiv:2210.04561, 2022.\n\n[2] Hansen N, Wang X. Generalization in reinforcement learning by soft data augmentation. ICRA 2021.\n\n[3] Hansen N, Su H, Wang X. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. NeurIPS 2021.\n\n[4] Yuan Z, Yang S, Hua P, et al. RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization. arXiv:2307.10224, 2023.\n\n[5] Schwarzer M, Anand A, Goel R, et al. Data-efficient reinforcement learning with self-predictive representations. ICLR 2021.\n\n[6] Li X, Shang J, Das S, et al. Does self-supervised learning really improve reinforcement learning from pixels? NeurIPS 2022.\n\n---\n**W2-1:** *\"[...] it would be valuable to compare Rand PR with PadCrop or Translate, examining factors such as hardness or spatial diversity.\"*\n\n**A**: Rand PR demonstrates superior augmentation effects for sample-efficient VRL compared to PadCrop on the vast majority of tasks in DMC and CARLA of our paper, and PadCrop has already been proven to be a better DA type than Translate, Rotate, etc [7]. Next we will illustrate Rand PR's advantages from the perspective of Hardness and Spatial Diversity.\n\n1. **Hardness:** Intuitively, by avoiding the inevitable edge information loss caused by other augmentation methods, Rand PR should achieve a lower level of hardness. To validate this, we experimentally measured the hardness of Rand PR and other DA methods including PadCrop on the CartPole Balance task. We pre-trained a clean policy on the unaugmented CartPole Balance task, then tested the performance of this policy under different augmentations and calculated the Hardness according to its definition. The results in the table below confirm our intuition: Rand PR induces markedly lower hardness compared to PadCrop and other DA types. Therefore, the intrinsically lower hardness of Rand PR contributes to its superior sample efficiency over other augmentations.\n\n    |Aug Type|Rand PR|PadCrop|Translate X|Translate Y|Rotate|\n    |:-:|:-:|:-:|:-:|:-:|:-:|\n    |Hardness|$1.33 \\pm 0.44$|$1.73 \\pm 0.55$|$2.17 \\pm 0.64$|$2.29 \\pm 0.80$|$2.65 \\pm 0.67$|\n\n2. **Spatial Diversity:** Rand PR offers sufficient spatial diversity through its large degrees of freedom in scaling ratio and content location. Compared to PadCrop and Translate, Rand PR introduces not only positional transformations, but also varying scaling ratios, in addition to positional transformations. How to quantitatively compare the spatial diversity of different augmentation methods is an open question worth further investigation in the future.\n\n[7] Kostrikov I, Yarats D, Fergus R. Image Augmentation is All You Need: Regularizing Deep Reinforcement Learning from Pixels. ICLR 2021. \n\n---\n**W2-2:** *\"[...] evaluating CycAug against other fusion schemes in terms of type diversity or data stability would provide further insights.\"*\n\n**A**: The type diversity of multi-type DA fusion schemes depends on the components being fused, thus it is infeasible to directly compare the type diversity across different fusion schemes. However, data stability when using the same DA components is a good way to characterize different fusion schemes' properties. To compare data stability, we evaluated the variance of Q values and Q-Target values during training when applying CycAug and other fusion methods on the Quadruped Run task. As shown in `Figure 2 of the Response PDF`, CycAug consistently induces lower Q value variance, empirically demonstrating its superior stability. A more stable training process contributes to CycAug's better sample efficiency."}, {"review_id": "Wfu1WNsxYI", "reviewer": "Reviewer_oGs4", "summary": "This paper explores the crucial attributes of domain adaptation (DA) in achieving sample-efficient visual reinforcement learning (RL) and emphasizes the specific requirements of DA for visual RL. Extensive experiments are conducted to investigate these attributes.\n\nThe paper introduces two practical guidelines that aim to maximize the potential of DA. The first guideline focuses on individual DA operations, while the second guideline explores multi-type DA fusion schemes. Based on these guidelines, the authors propose two improvement strategies, namely Rand PR and CycAug.\n\nCycAug, which incorporates Rand PR as a key component, is shown to outperform existing methods in terms of sample efficiency. Through comprehensive benchmark tasks on DM Control and CARLA, CycAug demonstrates state-of-the-art performance in enhancing sample efficiency in visual RL.", "questions": "What is different in below (i), (iii), (iv) on the page 2.\n\n(i) The training performance of visual RL is highly sensitive to the increase of DA’s hardness \n\n(iii) Unlimited increases in strength diversity can harm visual RL performance.\n\n(iv) Despite the increased type diversity, naively applying multi-type DA to visual RL training can lead to decreased performance.", "limitations": "- The problem of hardness from DA in RL has already been addressed, so there is no novelty in tackling it again.\n- It is necessary to also discuss the case of using more augmentation methods in CycAug.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The hardness of DA in RL has been well analyzed.\n\n2. Rand Padresize: This paper introduces Rand Padresize as a novel augmentation method in reinforcement learning (RL). Unlike cropping, Rand Padresize retains all the information, which is highly advantageous. This unique approach addresses the challenge of hardness from augmentation in RL, making it a notable strength of the paper.\n\n3. CycAug: The paper proposes CycAug, which employs a cycling method to overcome the potential disruption caused by excessively frequent variations. This demonstrates the authors' thoughtful consideration of the impact of augmentation strategies on the learning process. The inclusion of CycAug as a solution to mitigate disturbances is another strength of the paper.", "weaknesses": "1. Lack of novelty: The paper acknowledges the problem of hardness in RL as a well-known issue. \n2. Only use two augmentation in CycAug. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and insightful feedback. We will address each of your comments and concerns below and also in our revised manuscript.\n\n----\n**Q1**: *\"What is different in below (i), (iii), (iv) on the page 2. [...]\"*\n\n**A**: These three key findings correspond to the three sets of comparative experiments we conducted in Section 3.2 of the paper. While interconnected, each of the three findings offers distinct insights into constructing efficacious data augmentation for sample-efficient visual RL:\n\n(i) Compared to other domains like supervised learning, the training process of visual reinforcement learning is more sensitive to increases in the hardness of data augmentation (DA). Through quantitative experiments, we find even minor rises in hardness significantly impair training performance. This underscores the unique requirements of visual RL for DA design.\n\n(iii) While unlimited increases in strength diversity generally improve robustness in supervised learning and adversarial training, our experiments reveal this can actually harm performance in visual RL. While predominantly ascribed to the heightened hardness-sensitivity of reinforcement learning, this revelation likewise precipitates the judicious tuning of intricate data augmentation particulars to cater to the distinct demands of RL.\n\n(iv) Although elevating type diversity is widely deemed effective for enhancing DA, we discover directly applying multi-type DA fusion schemes from other fields fails to improve sample efficiency in visual RL. This abnormal failure can be partly ascribed to the sensitivity of visual RL to the hardness of DA, but is also largely due to training instability from complex transformations or dynamic fluctuations in the data distribution that result from frequent switching between different types of DA operations. We demonstrate in `Figure 2 of the Response PDF` that the original fusion schemes introduce high variance into the Q value estimates during training.\n\n---\n**Limitation 1:** *\"The problem of hardness from DA in RL has already been addressed, so there is no novelty in tackling it again.\"*\n\n**A**: To the best of our knowledge, there are mainly two studies that focus on the problem of hardness from DA in visual RL - SVEA[1] and SECANT[2]. However, these two methods are both proposed to enhance training stability when applying **strong data augmentation**, which is imperative for improving generalization in visual RL, but will heighten the difficulty of training. SVEA performs by (1) using only weak augmentation PadCrop (called random shift in its original paper) to compute Q-Target values, and (2) mixing data of observations augmented by strong augmentations with those augmented by only weak augmentations. SECANT opts to only use weak augmentation during RL optimization, and then distills the obtained policy into a student agent using a teacher-student framework, introducing strong augmentations with high hardness during the distillation process.\n\nThe core idea of these two studies is to leverage weak augmentation such as PadCrop (named as random shift in their orginal papers) to alleviate extra data hardness/instability introduced by strong augmentation. However, the investigation conducted in `Section 3.2` of our paper demonstrates that even for weak augmentations, changes in their hardness level still significantly impact sample efficiency. Therefore, the problem of hardness from weak DA in visual RL remains an unresolved issue warranting further efforts.\n\nIn this paper, we provide fine-grained analyses of the impact of hardness on training and find that even minor increases in hardness can have a significant negative impact on training performance. Based on these insights, we designed an augmentation method called Rand PR that has lower hardness compared to PadCrop, the most prevalent augmentation used in current visual RL methods. Our comprehensive investigation into the impact of DA hardness on training sample efficiency elucidates the unique requirements of visual RL for DA and provides actionable guidelines for design. These contributions are substantial and non-trivial.\n\nAdditionally, we innovatively conducted a systematic study on the impacts of three different diversity aspects (strength, spatial, and type) on sample efficiency in visual RL. The obtained conclusions provide actionable guidelines for designing DA operations suitable for visual RL scenarios.\n\n[1] Hansen N, Su H, Wang X. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. NeurIPS 2021.\n\n[2] Fan L, Wang G, Huang D A, et al. Secant: Self-expert cloning for zero-shot generalization of visual policies. ICML 2021.\n\n---\n**Limitation 2:** *\"It is necessary to also discuss the case of using more augmentation methods in CycAug.\"*\n\n**A**: CycAug is a multi-type DA fusion scheme that can handle more DA types, as long as these DA operations themselves have sufficient individual effectiveness. The reason we chose Rand PR and PadCrop as components for CycAug in the original paper is that these two augmentation methods demonstrate markedly superior effectiveness compared to other augmentations. However,  CycAug can be further expanded by incorporating more DA operations beyond just Rand PR and PadCrop.\n\nWe conducted further experiments on the Quadruped Run task to demonstrate the effects of CycAug with three and four components, as shown in the table below. Training curves are illustrated in `Figure 3 (Left) of the Response PDF`. Three-component CycAug utilizing PC, PR, and CS attained superior sample efficiency versus the dual-component CycAug (PC+PR) presented in our paper, exhibiting the capacity for additional expansion of CycAug.\n\n|Aug Component|Return|\n|-|-|\n|PC+PR | $728.6 \\pm 64$ |\n|CropShift (CS)|$536.5\\pm 89$|\n|Translate (Tr)|$467.4\\pm 9$|\n|PC+PR+CS|$783.9\\pm 46$|\n|PC+PR+Tr|$677.0\\pm 16$|\n|PC+PR+CS+Tr|$736.2\\pm 62$|"}, {"review_id": "hIspPknueK", "reviewer": "Reviewer_HheB", "summary": "This paper conducts a detailed study on what attributes of data augmention in Visual Reinforcement Learning are playing essetianal roles. With extensive experiments on DMC tasks, four main findings are given regarding the strength and diversity. Based on these findings, authors introduce two new data augmentation techniques: `Random PadResize` and `Cycling Augmentation` and achieves reasonble improvement over DrQ-v2.", "questions": "See `weaknesses` for questions.", "limitations": "The limitations have been mentioned by authors in the conclusion section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The attributes of data augmentation are well categorized and systematially studied, supported by extensive experiments.\n2. The newly proposed data augmentation mechanism is well motivated and simple enough for the Visual RL community to widely adopt.", "weaknesses": "1. **Lack of novelty.** The given properties of data augmentation are not too surprising and so is the proposed mechanism.\n2. **Limited performance.** As shown in Figure 8 and Figure 9, the sample efficiency is very close to DrQ-v2 (almost the same). However, the main claim made by authors is the improved sample efficiency. Moreover, from the curve trend in Figure 8 and Figure 9, a bit more training would possibly lead to the same performance of all these compared methods.\n3. **Lack of enough training.** The training steps across all environments are not enough to make the curves converge, thus it is unclear whether the newly proposed mechanism could lead to better convergence or merely give a mild improvement in limited steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and insightful feedback. We will address each of your comments and concerns below and also in our revised manuscript.\n\n----\n**W1**: *Lack of novelty.*\n\n**A**: We appreciate you raising the concern regarding novelty. While I understand your perspective that the proposed augmentation properties and mechanism may not seem overly surprising, we believe this work makes several novel contributions to the field of visual RL:\n\n1. our in-depth investigation helps fill a gap in analysis of DA attributes for visual RL. We provide quantitative experiments to how hardness, strength diversity, spatial diversity, and type diversity impact the effectiveness of DA in sample-efficient visual RL. This rethinking part of this paper provide insights to guide the design of RL-tailored DA techniques.\n2. The Rand PR individual augmentation incorporating controlled hardness and enhanced spatial diversity is an original contribution improving sample efficiency in visual RL.\n3. The CycAug multi-type fusion scheme tailored for RL via periodic cycling of diverse augmentations is also novel, benefiting type diversity while maintaining data distribution consistency.\n4. We achieve state-of-the-art sample efficiency on both DM Control and CARLA benchmarks, validating the efficacy of our proposed techniques.\n\n---\n**W2:** *Limited performance.*\n\n**A**: In this paper, our objective is to leverage the potential of data augmentation (DA) to enhance the sample efficiency of visual reinforcement learning (RL). This refers to achieving the highest possible performance within a limited number of iterations in the environment. As demonstrated in our paper through evaluations conducted in DMC and CARLA environments, as well as the supplementary experiments in the table-top manipulation environments of Robosuite (as shown in `Figure 1 of the Response PDF`), CycAug consistently achieves significantly higher sample efficiency than DrQ-V2 after a constrained number of interaction steps. Furthermore, when the allowed interaction steps are increased, CycAug exhibits faster convergence and higher final performance. For example in CARLA tasks, CycAug outperforms DrQ-V2 by substantial margins $23.8\\%$ in final performance and $43.7\\%$ in low data regime.\n\nA salient feature of the proposed Rand PR and CycAug techniques is their versatility as plug-and-play modules that can boost the effectiveness of existing algorithms. Our methods solely involve enhancing the data augmentation while keeping the base visual RL framework unchanged. As such, Rand PR and CycAug represent broadly applicable contribution modules, rather than complete stand-alone systems. The strengths of reduced hardness and controlled diversity in Rand PR, along with the stability benefits of cyclic augmentation in CycAug, can serve as universal plugins to augment prevailing RL methods. We empirically demonstrate their versatility by improving the state-of-the-art DrQ-v2. Furthermore, Rand PR and CycAug are compatible with and can potentially enhance myriad other cutting-edge visual RL algorithms. This underlines their value as versatile contribution techniques. \n\n---\n**W3:** *Lack of enough training.*\n\n**A**: Thanks for your suggestion. In order to ensure the convergence of algorithm performance, we increased the allowed training steps to twice their original values in four representative DMC tasks. We illustrate the average episode return after $3\\times 10^6$ frames in the below table. The complete training curves are depicted in `Figure 3 (Right) of the Response PDF`. CycAug achieves faster convergence than DrQ-V2 across all tasks and demonstrates higher final performance on Quadruped Run and Walker Run.\n\n| Task           |   | DrQ-V2 |   | CycAug |\n|----------------|---|--------|---|--------|\n| Quadruped Run  |   |$694.7\\pm 134$|   |$840.6\\pm 57$|\n| Walker Run     |   |$699.4\\pm 44$ |   |$772.6\\pm 17$|\n| Quadruped Walk |   |$920.3\\pm 13$ |   |$937.1\\pm 13$|\n| Hopper Hop     |   |$358.7\\pm 47$ |   |$366.5\\pm 65$|\n"}]

Author rebuttal:
# Global Response

---

Dear reviewers,

We are sincerely appreciative of the time and effort you dedicated to reviewing our manuscript. Your comprehensive feedback has offered us valuable insights for enhancing clarity and quality. We have individually responded to each reviewer's queries and suggestions. Here, we want to provide a few comments on the common concerns and highlight supplementary experiments included in the `response PDF`. We eagerly look forward to engaging in further discussions with reviewers to address any remaining concerns.

---
**Evaluation on more challenging visual RL tasks.**

In addition to the evaluations presented in the paper on DMC and CARLA, we further conduct evaluations on two challenging tasks, Lift and TwoArmPegInHole, in the table-top manipulation environments of **Robosuite**. We report the average episode return over 5 random seeds after training for 500k frames (with 2 action repeat) and present the complete 1M frames training curves in `Figure 1 of the Response PDF`. The experimental results demonstrate that CycAug achieves higher sample efficiency than the original DrQ-V2.


| Augmentation Type | Lift | TwoArmPegInHole |
|-------------------|------|-----------------|
| DrQ-V2            |$253.9\pm 125$|$295.2\pm 56$|
| CycAug            |$324.5\pm 120$|$350.4\pm 53$|

---
**Demonstration of data stability during CycAug training.**

To compare data stability, we evaluated the variance of Q values and Q-Target values during training when applying CycAug versus other fusion methods on the Quadruped Run task. Variance is consistently assessed through 4 forward passes, employing random data augmentation on identical observations. As shown in `Figure 2 of the Response PDF`, CycAug consistently induces lower Q value variance, empirically demonstrating its superior stability. A more stable training process contributes to CycAug's better sample efficiency.

---
**More DA components incorporated in CycAug**

CycAug is a multi-type DA fusion scheme that can incorporate far more than three DA types, as long as these DA operations themselves have sufficient individual effectiveness. We conducted further experiments on the Quadruped Run task to demonstrate the effects of CycAug with three and four components, as shown in the table below. Training curves are illustrated in `Figure 3 (Left) of the Response PDF`. Three-component CycAug utilizing PC, PR, and CS attained superior sample efficiency versus the dual-component CycAug (PC+PR) presented in our paper, exhibiting the capacity for additional expansion of CycAug.

|Aug Component|Return|
|-|-|
|PC+PR | $728.6 \pm 64$ |
|CropShift (CS)|$536.5\pm 89$|
|Translate (Tr)|$467.4\pm 9$|
|PC+PR+CS|$783.9\pm 46$|
|PC+PR+Tr|$677.0\pm 16$|
|PC+PR+CS+Tr|$736.2\pm 62$|

---
**More training steps to compare the final performance**

In order to ensure the convergence of algorithm performance, we increased the allowed training steps to twice their original values in four representative DMC tasks. We illustrate the average episode return after $3\times 10^6$ frames in the below table. The complete training curves are depicted in `Figure 3 (Right) of the Response PDF`. CycAug achieves faster convergence than DrQ-V2 across all tasks and demonstrates higher final performance on Quadruped Run and Walker Run.


| Task           |   | DrQ-V2 |   | CycAug |
|----------------|---|--------|---|--------|
| Quadruped Run  |   |$694.7\pm 134$|   |$840.6\pm 57$|
| Walker Run     |   |$699.4\pm 44$ |   |$772.6\pm 17$|
| Quadruped Walk |   |$920.3\pm 13$ |   |$937.1\pm 13$|
| Hopper Hop     |   |$358.7\pm 47$ |   |$366.5\pm 65$|

Meta-review: The submission addresses the data augmentation aspect of visual RL. An experimental analysis study is done, and a few proposals for improved data augmentation strategies are made. The fact that the paper has an analysis angle that helps the community gain an understanding of existing training techniques is appreciated. The reviewers have a consensus to accept the paper (one is a borderline accept). The rebuttal was considered and generally appreciated by the reviewers who engaged in the last phase. The reviewers shared several questions and comments. In particular, a few suggestions for more informative studies that would disentangle the contribution of different components and questions about the significance of the improvements were raised. The authors are strongly recommended to address the comments in the camera ready.
Decision: Accept (poster)