Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates
Authors: Nicholas Corrado
Josiah P. Hanna
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
data augmentation
model-free
Primary area: reinforcement learning
TL;DR: We study why incorporating dynamics-invariant augmented data into model-free RL updates improves RL training, and we find that improvements are largely due to an increase in state-action coverage and a decrease in the replay ratio of learning data.

Abstract: Recently, data augmentation (DA) has emerged as a method for leveraging domain knowledge to inexpensively generate additional data in reinforcement learning (RL) tasks, often yielding substantial improvements in data efficiency.
While prior work has demonstrated the utility of incorporating augmented data directly into model-free RL updates,
it is not well-understood when a particular DA strategy will improve data efficiency.
In this paper, we seek to identify general aspects of DA responsible for observed learning improvements.
Our study focuses on sparse-reward tasks with dynamics-invariant data augmentation functions, serving as an initial step towards a more general understanding of DA and its integration into RL training.
Experimentally, we isolate three relevant aspects of DA: state-action coverage,  reward density, and the number of augmented transitions generated per update (the augmented replay ratio).
From our experiments, we draw two conclusions: (1) increasing state-action coverage often has a much greater impact on data efficiency than increasing reward density, and (2) decreasing the augmented replay ratio substantially improves data efficiency.
In fact, certain tasks in our empirical study are solvable only when the replay ratio is sufficiently low.

Published as a conference paper at ICLR 2024
UNDERSTANDING WHEN DYNAMICS-IN DATA AUGMENTATIONS BENEFIT MODEL-FREE REINFORCE- MENT LEARNING UPDATES
Nicholas Corrado
Josiah P. Hanna
Department of Computer Sciences University of Wisconsin Madison ncorrado&wi edu
Department of Computer Sciences
University of Wisconsin Madison
iphannalca wisc. edu
ABSTRACT
Recently, data augmentation (DA) has emerged as a method for leveraging domain knowledge to inexpensively generate additional data in reinforcement learning (RL) tasks, often yielding substantial improvements in data efficiency. While prior work has demonstrated the utility of incorporating augmented data directly into model-free RL updates, it is not well-understood when a particular DA strategy will improve data efficiency. In this paper, we seek to identify general aspects of DA responsible for observed learning improvements. Our study focuses sparse- reward tasks with dynamics-invariin -nvvriiinn data augmentation functions, serving as an initial step towards more general understanding of DA and its integration into RL training. Experimentally, we isolate three relevant aspects of DA: state-action coverage, reward density, and the number of augmented transitions generated per update (the augmented replay ratio). From our experiments, we draw two conclusions: (1) increasing state-action coverage often has a much greater impact on data efficiency than increasing reward density, and (2) decreasing the augmented replay ratio substantially improves data efficiency. In fact, certain tasks in our empirical study are solvable only when the replay ratio is sufficiently low.
INTRODUCTION
Reinforcement learning (RL) algorithms are often data inefficient and often produce policies that fail to generalize outside of a narrow state distribution. Recently, number of RL algorithms and applications have been published that leverage data augmentation to enhance convergence and gener- alization (Mitrano and Berenson, 2022; Pitis et al., 2020; Qiao et al., 2021). Data augmentation (DA) is a technique in which agents generate additional synthetic experience by applying transformations to their observed experience. Since augmented data can be generated without the expense additional interaction with the environment, is attractive technique for improving the data efficiency of RL algorithms (Le., the number of environment interactions needed to solve a task).
Much of the prior work in DA for RL (Hansen and Wang, 2021; Laskin al., 2020; Raileanu et al., 2021; Wang et al., 2020; Yarats et al., 2020; 2021) builds off of DA techniques used in computer vision (Chen al., 2020). Other works have used domain-dependent DA strategies for non-visual tasks (Abdolhosseini et al., 2019; Mikhail Pavlov and Plis, 2018), including DeepMind's AlphaTTensor (Fawzi et al., 2022) which uses RL to discover more efficient matrix multiplications. These works introduce methods for generating augmented data and frameworks for integrating it into RL that demonstrably improve training performance. To the best of our knowledge, most prior work on DA has focused on introducing new types of data augmentation functions and demonstrating that they can boost the data efficiency of RL. What missing from the literature is a clear understanding of which aspects of DA yield improvements. Rather than adding to existing work by introducing new DA strategies, our main contribution an investigation into the following question:
When and why does data augmentation improve data efficiency in reinforcement learning?

---

Published as a conference paper at ICLR 2024
As a motivating example, consider using off-policy RL algorithm to solve a 2D navigation task in which an agent must reach random goal position (Fig. la). In this task, transitions observed by the agent can be augmented through either random translations of the agent (Fig. 1b) or random rotations of the agent and goal (Fig. lc). As shown in Fig. 1d, if we double the agent's learning data via DA and double the batch size used for updates, we achieve significant improvements in data efficiency com- pared to learning without DA. Furthermore, agents that learn from extra augmented data even surpass the perfor- mance of agents that learn from an equal amount of extra real data collected through additional environment inter- actions. More concretely, double the amount of policy data collected between updates and double the batch size used for updates that aon-auemented agents learn from the same amount of data and perform the same number of updates as the augmented agents. As shown in Fig. Id, additional augmented data leads to faster learning than simply collecting an equal amount of additional data from the agent's 's policy. In fact, doubling the learning data via nearly as good as learn-
*
(a) Observed
(b) Translate
(c) Rotate
Goal2D-v0
IQM Sucress Rate
1.0
0.5
policy
polioy dita
pallity date
X2 (003) via rocato
x2 datia via tiarolato
0.0
0.0
0.5
1.0
1.5
2.0
2.5
Updates
(d) Training curves
Figure 1: Visualizations of two augmen- tations translation (Ib) and rotation (Ic) - for a 2D navigation task in which an agent (black dot) must reach a goal (gold star). In 1d, "x.N policy data" cor- responds to collecting N times as many transitions with the agent's 's current pol- icy between updates, and "x2 via ro- tate/translate" corresponds to generating one augmented transition per observed transition. We increase the batch size and replay buffer sizes proportionally to the amount of extra data to keep the re- play ratio and replay age fixed across all experiments. We plot the interquartile mean success rate over 50 seeds with 95% bootstrap confidence belts.
the translation augmentation ing from 8 times as much anlicy-generater data. Thus, these augmentations must offer benefits beyond what addi- tional pollcy-ggneratee data can offer. An understanding of which aspects of DA yield these benefits will serve as an initial step towards guiding practitioners on how to more effectively incorporate DA into RL.
DA has taken many forms in the RL literature (Andrychow- icz et al., 2017: Hansen and Wang, 2021; Laskin et al., 2020; Pitis al., 2020; Qiao et al., 2021), and compre- hensive analysis of different DA frameworks, tasks, and data augmentation functions is beyond the scope of sin- gle study. Thus, in this work, we instead aim to better understand the benefits of integrating lynamiccs-nvvarian augmented data directly into model-free off-policy RL updates. With this focus in mind, we must leave studies on DA frameworks with auxiliary tasks (Hansen and Wang, 2021; Hansen et al., 2021; Raileanu et al., 2021; Wang et al., 2020), and studies on data augmentation functions that generate unrealistic data such visual data augmentations (Laskin et al., 2020) for future work.
Our investigation focuses on three aspects of DA that we hypothesize influence learning in sparse- reward tasks: an increase in state-action coverage via DA, the amount of additional reward signal generated via DA (reward density). and the number of augmented transitions generated per update (the augmented replay ratio). State-action coverage and reward density relate to how DA affects the agent's i distribution of learning data, whereas the augmented replay ratio relates to how augmented data is incorporated into RL training. We empirically ablate the effects of these factors using a simple and controllable DA framework similar to frameworks found in existing work (Laskin al., 2020; Pitis et al., 2020). In summary, our contributions are:
1. We introduce a framework for studying DA in RL that is amenable to analysis.
2. While is widely understood that high state-action coverage and discovery of reward signal are critical to data efficient RL, our experiments show that increasing state-action coverage via DA often has a much greater impact on data efficiency than increasing reward density.
3. We show that the success of DA depends strongly on the augmented replay ratio. In fact, certain tasks in our empirical study are solvable only when the augmented replay ratio is sufficiently low.
Code available htt.a//.w........ooooaianinnnnna
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
In this section, we provide an overview of data augmentation techniques and applications in RL.
Dynamics-basee Augmentation: Several prior works use data augmentation functions that affect the agent's 's current state, action, and next state. Pitis et al. (2020; 2022) stitch together locally independent features of different transitions generate additional data and provides a method for identifying local independence. Many model-based algorithms learn from synthetic data generated by a learned dynamics model and can be viewed as DA methods (Gu et al., 2016; Racanière et al., 2017; Sutton, 1990; Venkatraman et al., 2016).
Hindsight Experience Replay: In goal-conditioned RL, Hindsight Experience Replay (HER) (Andrychowicz et al., 2017; Fang et al., 2018; Liu et al., 2019; Rauber et al., 2017) counter- factually relabels the goal of a trajectory to generate additional data. This technique can be applied when transition dynamics are independent of the agent's ' goal, as is often the case. Follow-up work on HER has demonstrated that hindsight bias caused by changing the distribution of observed goals many hinder learning (Lanka and Wu, 2018; Li et al., 2020).
Applications of Domain-Specific Data Augmentation: Several recent works have leveraged domain-knowledger to create new data augmentation functions. Mikhail Pavlov and Plis (2018) and Abdolhosseini et al. (2019) apply DA to locomotion problems in which an optimal policy has a symmetric gait, and Mitrano and Berenson (2022) focus on augmenting trajectories of poses and movable objects relevant in robot manipulation. Qiao et al. (2021) consider DA in the context of differentiable simulation generate additional approximately correct transitions (a method they refer to as sample enhancement). Deep.Mind's ' AlphaTensor (Fawzi et al., 2022) exploits two invariances: tensor decompositions are commutative, and tensor rank is invariant the ordering of rows and columns. They exploit commutativity by generating additional augmented transitions and rank invariance using a network that disregards the row and column ordering of input tensors.
State-based Augmentation: Much of the prior work in DA for RL focuses on augmenting visual observations (Guan et al., 2021; Wang et al., 2020; Yarats et al., 2021). Laskin al. (2020) train RL agents on multiple views of visual states (crops, recolorations, rotations, etc.). Raileanu et al. (2021) introduce regularizers to ensure an agent's 's policy and value function are both invariant under augmentation. Sinha et al. (2022) ensure small perturbations of visuvl states state have similar state-action values. Hansen and Wang (2021) learn a state representation that is invariant under augmentation rather than directly using the augmented data for policy optimization. Hansen et al. (2021) identify sources of instability when performing visual DA. This line of work relates to domain randomization (Peng et al., 2018; Tobin et al., 2017), agents are trained to be robust randomized augmentations of observations. Visual augmentations are beyond the scope of our study; we focus on integrating augmented data that respects the environment's dynamics into model-fiee -reee updates, but visual augmentations generate unrealistic data and are typically used for auxiliary representation learning tasks. We provide further discussion on visual augmentations in Appendix A.
Invariant Model Architectures: DA often - ttouugh not always exploits known invariances within the environment's state space and/or dynamics. In this case, alternative DA is to simply hard- code these invariances into the agent's policy model (van der Pol et al., 2020; 2021; Wang et al., 2022). Residual Pathway Priors (RPPs) (Finzi al., 2021) capture invariances using a soft prior, biasing agents toward invariant policies without constraining them.
While these prior works focus on developing data augmentation functions or methods for incorporating augmented data into RL training, our work introduces a framework to investigate when and why DA improves learning.
PRELIMINARIES
In this section, we formalize the RL setting and the class of data augmentation functions we use.
3.1 REINFOR LEARNING
We consider finite horizon Markov decision processes (MDPs) (Puterman, 2014) defined by (8, A. p. do, 7) where S and A denote the state and action space, respectively, p(s' 8, a) denotes
3

---

Published as a conference paper at ICLR 2024
the probability density of the next state s' after taking action in state and r(s, a) denotes the reward for taking action a in state s. We write dy as the initial state distribution, y E [0. 1) as the discount factor, and H the length of an episode. We consider stochastic policies xo 8 A + [0, 1] parameterized by 0. The RL objective is to find a policy that maximizes the expected sum of discounted rewards (() Mo
Ei a.)
3.2 DATA AUGMENTATION FUNCTIONS
In the literature, data augmentation functions (DAFs) have taken different forms and served different purposes, We introduce a few important definitions to help classify the DAFs we focus on. Definition 1. A transition (s, a,r,s') is valid if it is possible under the transition dynamics and reward function, iee. p(s'
a) > O, and r r(s, a).
Definition 2. Let T C A R S denote the of possible transitions and let A(T) denote the set of distributions over T. A data augmentation function (DAF) is a stochastic function / T - A(T)
mapping transition (s, a. x')
augmented transition (5, à, (,,',)))) 8.).
Definition 3. A DAF is Synamicx-invariant if is closed under valid transitions.
We focus dynamics-invarian learning from data that does not match the MDP's dynamics can harm learning (Moerland et al., 2023). This focus includes methods such as HER (Andrychowicz al., 2017) and CoDA (Pitis et al., 2020) which provide gomain-independen DAFs. However, we do not restrict ourselves to domain- inmiinnnddeeendentt DAFs as, practice, domain-experi may be able to produce dynamics-inwe invariatt DAFs even though they cannot identify an optimal domain policy (Abdolhosseiní et al., 2019; Fawzi et al., 2022; Mikhail Pavlov and Plis, 2018). Our focus does exclude some recent works on DA especially those focusing visual augmentations (Laskin et al., 2020; Raileanu et al., 2021) that produce augmented states that would never be observed in simulation such that p(5' s, a) = O, since these augmentations do not satisfy our definition of valid. We note that dynamics-invariant DAFs will not necessarily preserve transition probabilities tasks with stochastic dynamics. We elaborate on the widespread applicability of dynamics-invariant DAFs in Appendíx A.
DAFs so that augmented data agrees with the underlying MDP, since
4 A FRAMEWORK FOR STUDYING DATA AUGMENTATION IN RL
Algorithm 1 Off-Policy RL with Data Augmentation
Prior work on DA in RL not only considers different DAFs but also various methods for integrating the augmented data into RL algorithms. To focus our study, we intro- duce a specific framework (Algorithm I) for incorporating augmented data into the training loop of any off off-policy RL algo- rithm. An off-policy algorithm is essential, augmented data may not be distributed according to the state-action distribution the current policy. We focus on the ef- fects integrating augmented data directly into policy and/or value function updates.
Data augmentation function f. augmentation ratio m, update ratio a, batch size b Initialize policy Te. replay buffer R, and augmented replay buffer fort 1,2.... do
Inputs:
Collect transition (8,, at: re, S1+1) using xæ Append (8t, r,, 8(+1) R
rt,
for i 1, 1,...,, do [8t, ae (1, Append (8r, ā., Fr, 8:41) ) to R
",,
if update then
Sample mini- mini-batch of b transitions from R Sample mini-batch 7 of ba transitions from R Update policy and/or value function using DUD
Within our framework, the agent observes a transition, applies a given DAF that transition to generate some number of aug- mented transitions, and then stores the ob- served and augmented transitions in separate replay buffers the observed and augmented replay buffers, respectively. When performing policy and/or value function updates, the agent samples data from both buffers and combines the data for the updates. To control how much augmented data generate and use in each update, we introduce a few parameters into the framework.
The augmentation ratio, m, specifies the number of augmented transitions generated per observed transition. Some DAFs, such as the translation augmentation Fig. 1b, can produce multiple unique augmentations from the same input transition. Each time the agent observes one real transition,
4

---

Published as a conference paper at ICLR 2024
m augmented transitions are sampled from the DAF. We use this parameter to study whether it is beneficial to produce multiple augmentations to diversify the augmented replay buffer. When the augmentation ratio is increased, we increase the augmented replay buffer size proportionally such that the age of the oldest observed and augmented transitions remain equal.
The update ratio, a, denotes the the ratio of augmented to observed data used for updates, e.g., a = 1 denotes that half of the data used for each update is augmented data. With access to large amounts of augmented data, it may be beneficial to increase the amount of augmented data used in updates. However, a large update ratio may exacerbate the tandem effect (Ostrovski et al., 2021), a decrease in performance when learning predominantly from data not collected by the agent.
The augmentation ratio modulates a third relevant quantity: the number of updates per augmented transition generated, or augmented replay ratio. 3 In the absence of DA, Fedus al. (2020) found that found that decreasing the replay ratio of observed data (the observed replay ratio) can improve data efficiency, though other works have improved data efficiency by developing techniques that enable learning with large replay ratios (Chen et al., 2021; D'Oro et al., 2023; Nikishin et al., 2022; Scheller et al., 2020). One can decrease the augmented replay ratio by increasing the augmentation ratio (m) while keeping the frequency of policy and/or value function updates fixed.
Though a variety of methods exist for incorporating augmented data into RI., our framework offers several core benefits for our study:
1. Control:
One can easily control the replay ratio and update ratio. Having control over the update ratio is especially important to ensure the agent uses a sufficient amount of observed data in each update mitigate the tandem effect. Moreover, since augmented data is stored in replay buffer and not sampled online, it is possible to keep the replay ratio of the augmented data equal to that of the observed data.
2. Lower Systematic Variance: Each update uses the same ratio of augmented observed data (i.e. update ratio), and the same number of augmented transitions are generated for every observed transition, eliminating a possible source of training variation.
3. Computational Efficiency: By storing and reusing augmented data, we improve computa- tional efficiency. While some DAFs can produce multiple unique augmentations of the same input transition, many can only produce a single augmentation - such as reflection in which case it is more efficient to reuse augmented data rather than generate new samples online every update.
Our framework is similar to those used in CoDA (Pitis et al., 2020), RAD (Laskin et al., 2020), and HER (Andrychowicz et al., 2017), as all three incorporate augmented data directly into updates without auxiliary tasks. We note that RAD as well as popular implementations (Raffin et al., 2021; Weng et al., 2022) of HER generate augmented samples during updates and discard them after use, whereas we save augmented data for reuse. Since we focus on using augmented data for model-free updates, Algorithm 1 not intended to capture methods that use augmented data for auxiliary tasks but easily extends to include such methods (Hansen and Wang, 2021; Raileanu et al., 2021).
5 DISENTANGLING PROPERTIES OF DATA AUGMENTATION
We identify aspects of DA that we hypothesize may impact its effectiveness within our framework.
State-Action Coverage: DAFs can generate data that the current policy otherwise might not observe, increasing state-action coverage. Greater state-action coverage via DA may aid exploration. However, it may also generate data that is very off-policy with respect to the current policy and hence increase the variance of learning due the tandem effect (Ostrovski et al., 2021).
"The age of a transition is the number of gradient steps taken by the agent since that transition was generated (Fedus et al., 2020).
"Prior works (D' (D'Oro al., 2023; Fedus et al., 2020; Nikishin al., 2022: Scheller al., 2020) define the replay ratio as the number of updates per environment interaction, characterizing how much the agent learns from existing data versus new experience. However, augmented data is generated by DAF and can produce multiple augmentations per observed transition. Thus, for our analysis, the number of updates per augmented transition generated is more appropriate metric.
5

---

Published as a conference paper at ICLR 2024
Long horizon, sparse reward tasks notoriously difficult since an RL agent is unlikely to discover reward signal through random exploration. A DAF that can produce transitions with additional reward signal could improve data efficiency. However, it is also known that reward- generating DA strategies such as HER (A (Andrychowicz et al., 2017) can bias learning and lead to overestimation of state-action values (Lanka and Wu, 2018) For sparse reward tasks, define reward density as the fraction of transition data in both observed and augmented replay buffers which successfully solve the task and thus contain reward signal."
Reward Density:
Augmented Replay Ratio: Some DAFs (such as the translation DAF in Fig. 1b) can generate multiple augmented transitions given a single input transition, substantially increasing the amount of data available to the agent. We hypothesize that it may be beneficial to generate as many augmented transitions as possible to lower the augmented replay ratio (Fedus et al., 2020).
While it is widely understood that high coverage and discovery of reward signal are critical to solving sparse-reward RL tasks (Andrychowicz et al., 2017; Tang et al., 2017). the degree to which increasing state-action coverage and reward density via DA individually contribute to data efficient RL is less clear. These factors are be difficult to completely isolate; since the reward function r(s, a) depends on and a, altering reward density necessarily changes state-action coverage. In our experiments, we attempt to isolate all three aspects of DA to determine how much each affects data efficiency.
5.1 EXPERIMENTS
We focus our experiments on four sparse-reward, continuous action panda -gym tasks (Gallouédec al., 2021): PandaPush-v3, PandaSlide- v3,
and PandaFlip-v3 (Fig. 2), which we hence- forth refer as the Push, Slide, PickAndPlace, and Flip tasks, respectively. We consider two DAFs:
1. TRANSLATEGOAL: Relabel the goal with a new goal sampled uniformly at random from the goal distribution.
Figure 2: PandaPush- v3. A robotic arm must push block to a goal location.
2. MAANSLATEGON.PROXIMMM
Relabel the goal with a new goal sampled from the goal distribution. With probability the new goal is sufficiently close the object generate reward signal.
We also consider toy 2D navigation task Goal2D-v0 (Fig. la) in which agent must reach random goal. The agent receives reward +1 when it sufficiently close to the goal and reward -0.1 otherwise. Agent and goal positions are initialized uniformly at random. We consider three DAFs:
1. ROTATE: Rotate the agent and goal by 0 E (5
2. TRANSLATE: Translate the agent to random position.
3. TRANSLLATEPPOXIIA.... Translate the agent random position. With probability p. the agent's new position is sufficiently close to the goal to generate reward signal.
These DAFs offer an avenue to investigate the role of reward density and state-action coverage in DA. For instance, one modify reward density through in TRANSLATEGOALPROXIII
Moreover, these DAFs are extremely general can be applied to many tasks, e.g. most navigation tasks. We include full descriptions of each environment and DAF in Appendices and C, respectively. We use DDPG (Lillicrap et al., 2015) for Panda tasks and TD3 (Fujimoto et al., 201 8) for Goal2D. Further training details are Appendix G. We include experiments studying how all three factors affect an agent's generalization ability in Appendix E and include experiments studying the augmented replay ratio for dense reward MuloCo tasks (Brockman et al., 2016) in Appendix F.
5.1.1 BENCHMARKING DATA AUGMENTATION
We first benchmark the performance of DA against simply collecting more policy data to establish how much chosen DAFs improve data efficiency. Prior work has demonstrated that learning with augmented data is often more data efficient than learning without it, though it is unclear how learning
"ith dense rewards, one may need consider the full distribution of rewards in the replay buffer instead.

---

Published as a conference paper at ICLR 2024
policy data
policy data
x4 policy
data TransiateGoal
data TranslateGool
PandaPush-w3
PandaSiide-3
PandaPickkAnPPaazo
Paada--lipvv
ICM Success Rate
1.00
1.00
0.8
0.75
0.75
0.6
0.50
0.50
0.4
0.25
0.25
0.2
0.00
0.00
0.0
0.0
0
0
2
1e5 0.0
2.5
5.0
7.5
Updetes
te5
Updates
Updates
1e5
Updates 1e5
Figure 3: "x2 policy data" agents double their learning data by collecting twice as many samples between updates, and "x2 data via TranslateGoal" agents double their learning data by generating one augmented transition per observed transition. Note that the horizontal axis shows the number of updates rather than timesteps. Each curve shows the interquartile mean over 10 seeds. Shaded regions denote 95% bootstrap confidence belts.
from augmented data compares to learning from additional policy-gennnn generated data, as data and augmented data are distributed differently in general. In these experiments, we increase the available learning data using DA or by collecting more data with the agent's 's current policy between updates. We label agents according to how many additional environment interactions they perform, e.g. "x2 policy data" corresponds to one extra environment interaction, and "x2 data via TRRANS.ATEGOOA." corresponds to generating one augmented transition per observed transition (m = 1). When collecting or generating extra data, we increase the batch size and replay buffer size proportionally so that all agents learn with the same amount of data, the same observed and augmented replay ratios, and the same replay age. Thus, augmented and observed data are treated equally in training.
From Fig. 3, we see that TRANSLATEGOAL offers significant improvement For instance, in Push, doubling the learning data via TRANSLATEGOAL doubles data efficiency. Though, additional policy-generates generated data generally yields equal or better performance. Having established that TRaNSLATEGOAL improves data efficiency, in the following sections, we study the degree to which reward density and state-action coverage are responsible for these improvements.
additional data.
5.1.2 STATE-ACTION COVERAGE
In this section, we study how increasing state-action coverage via DA affects data efficiency. Since the reward is function of the agent's state and action, it is diffccult to completely isolate the effects of in- creased coverage: change in coverage affects reward density. Nevertheless, we can better understand the effect of increasing coverage by comparing agents trained using TRANSLATE and TRANSLATE- and TRANSLATEGOALP PROXIMALL))) DAF's that increase state-action coverage without generating additional reward signal. Early in training when there is little to reward signal in the observed replay buffer, TRANSLATEGOALIOO and CEESSAATEGALPPROXIIE (0) have little affect on reward density. As the policy learns and more reward signal is added to the observed replay buffer, the lack of reward signal in the augmented replay buffer reduces the overall reward density. Thus, we can attribute any performance boost provided by
GOAL with agents trained using
an increase in state-action coverage and/or a decrease in reward density. To better separate the effects of increased coverage and decreased reward density, we double the agent's training data using different ratios of augmented data to observed data (1:5, 1:2, and 1:1). A smaller split of TRANSLATEPROX (AALOO/AAASLLTGGOOA PROXIMMA data corresponds decrease in reward density. We report results for ratios yielding the largest improvements to data efficiency (i.e. ratios that best balances the increase in coverage with the decrease in reward density).
and
less coverage but also a smaller
As shown in Fig. 4, WWW.SSAATTGGOA....OO.MAEA
in Slide, PickAndPlace, and Flip is as data effi- cient as TRANSLATEGOAL: increased coverage alone explains the benefits of LRANSSATEGOAL in and TRANSLATEGOAL AAOLSAAAAGGA
these tasks. In Goal2D and Push, PRANSLLATEPROXIMALL are more data efficient than no DA but less data efficient than TRANSLATE and TRANSLATEGOAL. Thus, although increased state-action coverage is the primary benefit in most tasks, we see that increased reward density can also play a role. In the following section, we further disentangle state-action coverage and reward density assess how critical high reward density is for these tasks.
7

---

Published as a conference paper at ICLR 2024
x1 poliyy 50ta tno augmoriatinn
Sorrsan TransansProeina00
TransoteGool /Transale
uah-0
PanduFiipvva
Baceess Rus.
1.00
2M Suctes Ral
1.00
0.4
0.70
0.75
0.75
0.6
0.00
0.50
0.50
0.2
0.20
0.25
0.25
0.1
0.00
0.00
0.0
Upiatex
1
Updiste
100
Upclstes
165
165
Upchraes 165
Figure 4: Learning with TRANSLATEGOAL, TRANSLATEPROXMAL(O)) We plot the IQM success rate over 10 seeds for Panda experiments and 50 for Goal2D, Shaded regions denote 95% bootstrap confidence belts.
TRANSLATE, and
xi policy data (ro aegmorratioo)
p-0
p- 0.05
p- 0.1
0.25
1.00
40
1.00
PandaPash-00
1,00
Panda686de-8
PanduFiipvv
22 Bacoes Ras
2M D44e
OMM sucoede Role
8.70
8.75
0.75
0.6
0.2
0.00
0.50
0.20
0.25
0.1
0.00
Upiatex 5
Updatte 105
Uptlstex
105
Loccsee 165
Upchrae 165
Figure 5: Learning with TRANSLATEGOAL PROXIMALL()))) for various p. Darker colors correspond to larger values. The dashed red line for PickAndPlace denotes the final IQM success rate achieved by in Fig. 4. We plot the IQM success rate over 10 seeds for Panda experiments and 50 for Goal2D. Shaded regions denote 95% bootstrap confidence belts.
CAANSLATEGOALPPOXIIM
5.1.3 REWARD DENSITY
We now strive further disentangle state-action coverage from reward density by studying how changes to reward density affect learning. In the following experiment, we modify reward den- sity by varying the probability p of generating reward signal with TRANSLATEEPRXXXMAA... and while keeping the update ratio and augmentation ratio fixed at a = 1 and m = 1, respectively. This setup does not completely isolate reward density from state-action coverage; changing reward density (increasing p) also affects state-action coverage, as increases the amount of learning data in which the goal is near the object. Nevertheless, this experiment enables us to answer the following question: how critical is it that DA generates data with high reward density?
As shown in Fig. 5, changing p has little effect data efficiency in Slide and Flip. further supporting that increased coverage is the primary benefit of TRANSLATEGOAL in these tasks. In Push and PickAndPlace, p 0.05 is most data efficient*, while in Goal2D, data efficiency increases signifi- cantly as with p = 0.01, and increasing to p 0.1 offers marginal additional improvement. In these three tasks, the largest p values decrease data efficiency, since changes the distribution of reward signal and can bias updates (i.e. hindsight bias (Lanka and Wu, 2018; Li et al., 2020)). Since the most data efficient learning occurs when DA contributes no reward signal or a relatively small amount, we conclude that high reward density is not critical to successful DA.
Collectively, our state-action coverage and reward density experiments suggest that increasing state- action coverage via DA often has a much greater impact on data efficiency than increasing reward densiry. Thus, our results suggest that RL practitioners choosing among candidate DAFs or designing new DAFs should focus increasing state-action coverage more so than increasing reward density.
5.1.4 AUGMENTED REPLAY RATIO
Our previous experiments study how two properties of DAFs affect RL, though performance may be sensitive to how augmented data is incorporated into RL training. In this section, we study how the number of updates per augmented transition generated affects data efficiency.
Existing DA strategies often incorporate multiple augmentations of the same observed transition into policy optimization. For instance, HER (Andrychowicz et al., 2017) generates 4 hindsight transitions
'In Fig. 4, TRANSLATTPROXIIAA GOAL achieves an IQM success rate of 0.3 (red dashed line), outper- forming all Pick AndPlace agents Fig. 5; high reward density augmented data is not critical in this task.
8

---

Published as a conference paper at ICLR 2024
polcy dara
TrandateGool, p
0.5
TrinsanGGoll 0025
TransoneCGoll 6 0.325
TransbneGGoa. P 0.0625
Pa
PandaSiide-33
PandaPPikkAndPaace--3
Panda-iip-3
1.00
1.00
1.00
IQM Success Rate
Rate
IMM Success Rate
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0:25
0.00
0.00
0.00
0.0
0.5 Updates
1.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
Updates
te5
1e5
Updates
1e6
Updates
1e6
Figure 6:
Decreasing the augmented replay ratio with the update ratio fixed at a = markedly improves learning. Darker colors correspond to smaller replay ratios. We plot the IQM success rate over 10 seeds. Shaded regions denote 95% bootstrap confidence belts.
per observed transition, and CoDA (Pitis et al., 2020) generates up to 16 augmentations per observed transition. Generating additional augmentations decreases the augmented replay ratio, the number of updates per augmented transition generated. We hypothesize that decreasing the augmented replay ratio can provide similar benefit to decreasing the replay ratio of observed data noted by Fedus et al. (2020). Decreasing the replay ratio of observed data can be expensive requiring more environment interactions between updates while decreasing the augmented replay ratio is comparatively cheap.
In this experiment, we decrease the augmented replay ratio by generating more augmentations per observed transition (ie., increasing m.) We scale the augmented replay buffer size proportionally and keep the ratio of augmented to observed data used in updates fixed at a = 2. As shown in Fig. 6, a lower augmented replay ratio alone substantially improves data efficiency and overall performance across all panda tasks. Moreover, a low replay ratio is necessary to solve PickAndPlace within our training budget; 100% success rate can be achieved with 3 0.0625. while no learning occurs with = 0.5. A lower replay ratio also increases data efficiency in Goal2D with both TRANSLATE and ROTATE DAFs; due to space constraints, replay ratio is a preferable alternative to increasing the amount of augmented data used in updates, as the latter may exacerbate the tandem effect (Ostrovski et al., 2021) which RL from passive data fails. We support this claim with additional experiments provided in Appendix D.I.
include these figures in Appendix D.3. Decreasing the
Empirically, we have demonstrated that a DAF's success may depend strongly on how integrate its augmented data into training. To effectively apply DA, one must understand desirable properties of DAFs and relevant implementation details.
CONCLUSIONS, LIMITATIONS, AND FUTURE WORK
While prior work has demonstrated that incorporating augmented data model-free off-policy reinforcement learning (RL) updates improve the data efficiency of RL algorithms, we lack a clear understanding of which aspects of data augmentation (DA) yield such improvements. In this paper, isolated three aspects of DA in sparse reward tasks with dynamics-invariant data augmentation functions (DAFs) state-action coverage, reward density, and the replay ratio of augmented data (the number of augmented samples generated per timestep) - to understand how each affects performance, Empirically, we showed how increasing state-action coverage often has a much greater impact on data efficiency than increasing reward density. Moreover, we demonstrated that the decreasing the augmented replay ratio can yield dramatic improvements to data efficiency. In fact, certain tasks are unsolvable unless the replay ratio is sufficiently small.
Our work has provided an imitial study analyzing the benefits of DA. To better leverage DA, further work should study (1) how other properties of DAFs influence RL training, such as relevancy (Mitrano and Berenson, 2022), (2) how hyperparameters within a DA framework affect performance, and (3) how different RL algorithms affect performance. Our analysis focused on relatively low-dimensional sparse reward tasks with continuous actions, and findings may differ for tasks with dense rewards, discrete actions, or high- high-dimensional visual observations. Since our DA framework only integrates augmented data into model- fodel-free updates, it would be beneficial to extend this analysis to frameworks that use augmented data for auxiliary tasks such as representation learning rather than - or in addition to policy optimization (Hansen and Wang, 2021; Raileanu al., 2021).

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Thanks to Brahma Pavse and to the anonymous reviewers for feedback that greatly improved our work. This research is supported part by American Family Insurance through research partnership with the University of Misconsin-Madison 's Data Science Institute and the Office of the Vice Chancellor for Research and Graduate Education the University of Wisconsin Madison with funding from the Wisconsin Alumni Research Foundation."
REFERENCES
Farzad Abdolhosseini, Hung Yu Ling, Zhaoming Xie, Xue Bin Peng, and Michiel Van de Panne. On learning symmetric locomotion. In Motion, Interaction and Games. 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. 2020.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In International Conference on Learning Representations, 2023.
Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. Dher: Hindsight experience replay for dynamic goals. In International Conference on Learning Representations, 2018.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Moham- madamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 2022.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Conference on Machine Learning, 2020.
Mare Finzi, Gregory Benton, and Andrew Wilson. Residual pathway priors for soft equivariance constraints. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, 2018.
Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen. panda-gym: Open- Source Goall-Conditioned Environments for Robotic Learning. 4th Robot Learning Workshop: Self-SSuprvised and Lifelong Learning NeurlPS, 2021.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q- gelaarning with model-based acceleration. In International conference machine learning, 2016.
L. Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human duman--uuded reinforcement learning with explanation and context-aware data augmentation. NeurlPS, 2021.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmen- tation. In 202/ IEFE International Conference Robotics and Automation (ICRA), 2021.
10

---

Published as a conference paper at ICLR 2024
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems (NeurIPS), 2021.
IEEE Spectrum. Kiva systems: Three engineers, hundreds of robots, warehouse.. 2008. URL https///peccruu..eee
www.imiiiie...o/trreeeeeeeeeeeee org/three-engineers- engineers- hundreds- cundreds-of-robeee of-rooots-oooeo oeeeoooe----aae warchouse. [Online; accessed September 11, 2023].
Innovation Matrix Inc. Fetch mobile manipulator robot, 2023. URL https://nnovvation moaatix.com//
imj-fetch sejj-fetch-manne manipmlator------ee wew-orrooottaaaeee annoucement/. [Online; accessed September 11, 2023].
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
Sameera Lanka and Tianfu Wu. ARCHER: aggressive rewards to counter bias in hindsight experience replay. CoRR, 2018.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Alexander Li, Lerrel Pinto, and Picter Abbeel. Generalized hindsight for reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: /500vv::500.000 htt.//009929999 2015.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. arXiv preprint 1902.000990 2019.
Sergey Kolesnikov Mikhail Pavlov and Sergey M. Plis. Run, skeleton, run: skeletal model physics-based simulation. AAAI Spring Symposium Series, 2018
Peter Mitrano and Dmitry Berenson. Data augmentation for manipulation. arXiv preprint
arXiv:2205.02886. 2022.
Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn Jonker, et al. Model-based reinforce- ment learning: A survey. Foundations and Trends@ in Machine Learning, 2023.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Puerre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning.
PMLR, 2022.
Georg Ostrovski. Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances Neural Information Processing Systems (NeurIPS), 2021.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference roboties and automation (ICRA), 2018
Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based coun- terfactual data augmentation. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, et al. Multi-goal reinforcement lear- ing: Challenging robotics environments and request for research. arXiv preprint arXiv: 100. 2018.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
11

---

Published as a conference paper at ICLR 2024
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Efficient differentiable simulation of articulated bodies. In International Conference on Machine Learning, 2021.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. innginattionaugmeen agents for deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. : Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning, Advances in Neural Information Processing Systems (NeurIPS), 2021.
Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Juergen Schmidhuber. Hindsight policy gradients. arXiv preprint 17Xiv:1111.06000 2017.
Christian Scheller, Yanick Schraner, and Manfred Vogel. Sample efficient reinforcement learning through learning from demonstrations in minecraft. In NeurIPS 2019 Competition and Demonstra- fion Track. PMLR, 2020.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, Proceedings of Machine Learning Research, 2022.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990. 1990.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2017.
Oregon State University. Bipedal robot developed at oregon state achieves guin- ness world record 100 meters, 2022. URL https:/todaay.rrgonsttttcc. /dd/////wttees hetpbooddeeeooop--oggnn-ttttttte wwwhhhewwwwgiiiiieeswweeeerorooe 100-meters. [Online; ac- cessed September 11, 2023].
Elise der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling, Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Elise van der Pol, Herke van Hoof, Frans A Oliehoek, and Max Welling. Multi-agent mdp homomor- phic networks. arXiv preprini arXiv:2 10.04499, 2021.
Arun Venkatraman, Roberto Capobianco, Lerrel Pinto, Martial Hebert, Daniele Nardi, and J Andrew Bagnell. Improved learning of dynamics models for control. In International Symposium on Experimental Robotics, 2016.
Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant q learning in spatial action spaces. In Conference on Robot Learning, 2022.
Dian Wang, Jung Yeon Park, Neel Sortur, Lawson L.S. Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry. In International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2024
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 2022.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference Learning Representations, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.00645, 2021.
13

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Dynamics-Invariant Data Augmentation Functions
15
B Primary Environments for Our Emprical Analysis
16
C Data Augmentation Functions
17
D Additional Experiments
18
Increasing the Update Ratio
D.2 Increasing the Batch Size
18
D.3 Goal2D Augmented Replay Ratio
19
E Generalization Experiments
20
E.1 State-Action Coverage
20
E.2 Reward Density
20
Augmented Replay Ratio
F
MuJoCo Experiments
22
F.1
Environment Modifications
22
F2
Data Augmentation Functions
23
F3
Augmented Replay Ratio
24
G
Training Details
25
14

---

Published as a conference paper at ICLR 2024
fetch
(a) Navigation
(b) Locomotion
(c) Manipulation
Figure 7: Common data augmentation functions. (7a, Amazon warehouse robots (2008)): If a robot moving free space, transition dynamics are often invariant the agent's 's position. (7b, OSU's Cassie robot (2022)) Since robots are often symmetric about their sagittal axis, we can reflect the robot's left and right movements. (7c, Fetch robot (2023)) Objects move only if the agent contacts Thus, if the agent and object are not in contact, their dynamics are independent.
A DYNAMICS-INVARIAN DATA AUGMENTATION FUNCTIONS
In this section, we further motivate our focus on dynamics-invarian data augmentation functions. Specifying a dynamics-invarian data augmentation function requires knowledge of domain-specific invariances symmetries. While domain knowledge may seem like limitation, we observe in the literature and real-world RL applications that such invariances and symmetries are incredibly common and often require very little prior knowledge to specifyy We provided few examples:
1. Transition dynamics are often independent of the agent's goal state (Andrychowicz et al., 2017).
2. Objects often have independent dynamics if they physically separated (Pitis et al., 2020; 2022), which implies that objects exhibit translational invariance conditioned on physical separation.
3. Several works focus rotational symmetry of 3D scenes in robotics tasks (Wang al., 2022; 2023), and many real-worid robots are symmetric in design and thus have symmetries in their transition dynamics (Abdolhosseini et al., 2019; Mikhail Pavlov and Plis, 2018).
We include real-world tasks that exhibit one or more of these invariances in Fig. 7. We choose to focus on synammic-invarrinn data augmentations because they have already appeared widely in the literature. As RL becomes an increasingly widely used tool, we anticipate that domain experts will be able to identify new domain-specific augmentations and them to further lower the data requirements of RL. These observations underscore the importance of identifying when and why different general properties of data augmentation will benefit RL.
While visual DAFs are also commonly used in RL, study on such DAFs is beyond the scope of this analysis. Such studies will likely need focus on different aspects of DA than the ones we considered in our work. Visual DAFs generate augmented data with the same semantic meaning as the original data, and this point has three implications:
1. It may be imprecise to say that visual DAFs increase coverage, because the observed and augmented data have the same semantic meaning.
2. Because the original and augmented observation have the same semantic meaning, the augmented reward will often be the same as the original reward. Thus, the concept of "reward density" does not apply to these augmentations.
3. Moreover, because these DAFs produce data which could never be observed through environ- ment interaction (c.g. agent would never receive cropped, recolored, and rotated image from the environment), they primarily aid representation learning. In contrast, dynamics- invariant DAFs aid exploration.
15

---

Published as a conference paper at ICLR 2024
(a) Push
(b) Slide
(c) PickAndPlace
(d) Flip
Figure 8: Renderings of various Panda tasks. In PandaPush-v3 and Slide, the agent must move
object to a goal position. In PandaFlip-v3, the agent must rotate an object to a goal orientation.
B PRIMARY ENVIRONMENTS FOR OUR EMPRICAL ANALYSIS
We use four tasks from panda grm (Gallouédec al., 2021) as the core environments in the main paper. Fig. 8 shows renderings for each task.
PandaPush-v3 (Push): The robot must push an object to a goal location on the table.
The goal and initial object positions sampled uniformly random from (r, v) [-0.15, 0.15] = 0.02.
PandaSlide-v3 (Slide): The robot must slide a puck to a goal location on the table. The initial object position sampled uniformly at random from [-0.15, 0.15)3, while the goal (z, y, 2) is sampled from x [0.25, 0.55], y E [-0.15, 0.15).
0.015.
The robot must pick up an object and move it to goal location. With probability 0.3, the goal is the table (a 0.02), and with probability 0.7, the goal is in the air (z (0.02, 0.2)).
PandaPickAndPlaaee
PPicAnnPlaae
PandaFlip-v3 (Flip): The robot must pick up an object and rotate to a goal orientation. The initial object position is sampled uniformly at random from [-0.15, 0.15]², while the goal a uniformly sampled orientation expressed a quaternion.
In Push, PickAndPlace, and Flip, the object is a cube with side length 0.04. In Slide, the object is a cylindrical puck with height 0.03 and radius 0.03. The object's coordinate measures the distance between the center of the object and the table (e.g., In Push, Slide, and PickAndPlace, = = 0.02 means the object is the table).
Push, Slide, and PickAndPlace share similar sparse reward structure. The agent receives a reward of 0 if the object is within 0.05 units of the goal and reward of -1 otherwise. In Flip, the agent receives a reward of if the object's orientation q is within 0.2 units from the goal orientation Qo under the following angle distance metric:
1 cos(
d(g. .g) = 1 (q 92
where 9 is the angle of rotation required to rotate q to 9g- Otherwise, the agent receives reward of -1.
In the toy 2D navigation task Goal2D, agent reach fixed goal within 100 timesteps. The agent's state (x. y, Jg My) contains the coordinates of agent's positions (æ. y) and the goal's position
(Igg Mg). At each timestep, the agent chooses an action (r, 0) and transitions new position:
1++1 3t. + 0.05r cos(()
+ 0.05r sin
Thus, the agent moves at most 0.05 units in any direction. The goal position is fixed throughout an episode. The agent receives reward +1 when it is within 0.05 units of the goal and reward -0.1 otherwise. Agent and goal positions are initialized uniformly at random in [-1,
16

---

Published as a conference paper at ICLR 2024
C DATA AUGMENTATION FUNCTIONS
In this section, we provide further details on the data augmentation functions introduced in Section 5.1.
TRANSLATEGOAL:
Goals are relabeled using a new goal sampled uniformly at random from the goal distribution. Reward signal is generated when the new goal is sufficiently close to the object's s current position. To approximate the probability of this augmentation generating reward signal in each task, we sample 10M object and goal positions uniformly at random and report the empirical probability of the goal being sufficiently close to the object to generate reward signal.
PandaPush-v3 (Push): Reward signal is generated with probability approximately 0.075.
PandaSlide-v3 (Slide):
The probability of generating reward signal depends on the current policy. The initial object and goal distributions are disjoint, so this augmentation only generate reward signal if the agent pushes the object into the region x [0.25, 0.55], E [-0.15, 0.15). If the object is in this region, this augmentation will generate reward signal with probability approximately 0.075.
PandaPickAnnPPleee... (PickAndPlace):: Reward signal generated with probability approximately 0.04.
PandaFlip--33 (Flip): Reward signal is generated with probability approximately 0.04.
Goals are relabeled using a new goal sampled from the goal distribution. With probability p. the new goal generates a reward signal, and with probability 1 p. reward signal is generated. When generating augmented sample with reward signal, the goal is set equal to the object's position plus a small amount noise. and with probabilityy 1 p, the object is moved to random location sufficiently far from the goal that no reward signal is generated.
All Panda augmentation functions relabel the goal and reward. For Goal2D, we consider three data augmentation functions:
1. TRANSLATE: Translate the agent random position in [-1,+1]3 This augmentation generates reward signal with probability approximately 0.019. We obtained this approxima- tion by sampling 10M agent and goal positions uniformly random and then computing the empirical probability of the goal being with 0.05 units of the agent.
2. ROTATE: Rotate both the agent and goal by E {n/2, x, 3x /2). When sampling mul- tiple augmentations of the sample observed transition, it is possible to sample duplicate augmentations.
3. TRANSSLATEPROXIMMA(D") Translate the agent to a random position in +1]2. With probability agent's new position is within 0.05 units of the goal and generates reward signal, and with probability 1 - p, agent's new position is more than 0.05 units from the goal and generates reward signal.
All Goal2d augmentations modify the agent's state TRANSLATE and TRANSLATEEROXIMAL.. modify the agent's position and reward but do not modify the goal. ROTATE affects the agent's position, the goal position, and agent's ' action, but does not change the reward.
17

---

Published as a conference paper at ICLR 2024
x3 polloy data
x3 data via TrannlateCoel
x5 data via TramelateCoal
x8 data via TranslateCaa
x17 deta vis TranslateCoal
Pa
Poe diaSiide-33
Parrceeiip-a
1.0
1.0
1.0
0.8
0.6
Succeen Rabie
Success Rate
Sucosss Rale
0.6
0.0
0.8
024
044
M
0.4
0.0
044
0.0
02
0.4
0.6 0.6 Updates
1.0
1.2
0.3
0.4
Updates
165
Updates
1e6
10e
Updates
100
(a) Push
(b) Slide
(c) PickAndPlace
(d) Flip
Figure 9: Increasing the update ratio while keeping the augmented replay ratio fixed at 3 = 1 may harm performance. We plot the mean over 10 seeds expect for agents that very large batch sizes: "x9 data" and "x17 data" curves show seeds; "x17 data" for PickAndPlace shows seeds. Shaded regions are 95% confidence belts. These agents are trained using the same hyperparameters those used in Fig. 6, though the hyperparameters are slightly different than those used in other figures. See Appendix G for further details.
PanndaPuch-s5 v3
Per Slide v5
Pandel Up
Pands@PckAndPPaae
0.5
0.8
0.5
0.8
Suzoose Rule
Success Rale
Succoss Rate
0.6
0.6
Succesa
124
0.4
0.4
0.1
0.0
00
0.5
1.0
1.5
2.0
26
3.0
Updates
Updaste
Updates
Updates
(a) Push
(b) Slide
(c) PickAndPlace
(d) Flip
Figure 10: Increasing the batch size without increasing the amount of learning data available to the agent harms performance. We train agents using the hyperparameters listed in Table 2 with various batch sizes, e.g., "x2 batch size" corresponds to learning with batch size twice as large the batch size listed in Table 2.
D ADDITIONAL EXPERIMENTS
In this appendix, we provide additional experiments supporting the core claims in the main paper.
D.1 INCREASING THE UPDATE RATIO
In Section 5.1.4, we demonstrate that generating more augmented data to decrease the augmented replay ratio can drastically improve data efficiency. we generated more augmented data by increasing the augmentation ratio, we could alternatively incorporate the additional augmented data by using more augmented data in each update (i.e., increasing the update ratio). Fig. 9 shows agent performance the augmentation ratio and update ratio increase proportionally. We additionally keep the replay age fixed by increasing the augmented buffer size proportionally. Learning is generally more data efficient with a larger update ratio, though it may harm performance Notably, the improvements in data efficiency from decreasing the replay ratio (Fig. 6) are similar or better than those produced from an increased update ratio and can be achieved much lower computational cost per update.
in Slide.
D.2 INCREASING THE BATCH SIZE
In Fig. 3, agents with more training data available to them use larger batch sizes for updates, giving these agents a seemingly unfair advantage over agents that learn from less data. However, Fig. 10 shows that increasing the batch size without increasing the amount of data available to the agent
18

---

Published as a conference paper at ICLR 2024
Goal2D-v0
Goal2D-v0
1.0
1.0
0.8
0.8
Success Rate
Success Rate
x2 policy data
poiccy data
0.2
Tranelale, 8
0.2
Rotale, 0.5
Translale, 8 0.25
Rolale, 8 = 0.25
Transiale, B 0.0625
Rotale, 8 0.0625
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
Updates
165
Updates
165
(a) Goal2D: TRANSLATE
Goal2D: ROTATE
Figure 11: (50 seeds) Decreasing the replay ratio while keeping the update ratio fixed at a 1 improves data efficiency.
harms performance, due increase in the expected number of times transition is sampled for gradient update (Fedus et al., 2020). By scaling the batch size with the amount available learning data in Fig. 3, we keep the expected number of gradient updates per observed/augmentte transition fixed across all experiments, providing a fairer comparison.
D.3 GoAL2D AUGMENTED REPLAY RATIO
As in Section 5.1.4, we decrease the augmented replay ratio 3 by generating more augmentations per observed transition. We scale the augmented replay buffer size proportionally and keep the ratio of augmented to observed data used updates fixed at a 1. As shown Fig. 11, lower augmentation replay ratio increases data efficiency.
19

---

Published as a conference paper at ICLR 2024
Coal2D-v0
PandaPush-v3
PandaSlide--@
1.0
1.0
1.0
1QM Succegs Rale
0.8
0.2
L.T.
1QM
DLCh Terekahhiiee hoceeic
0.0
0.0
0.0
0.0
0.5
1.5
20
255
0.0
0.5
2.0
2.5
3,0
3
Updates
Updates
Updates
(a)
(b)
(c)
Figure 12: Learning with various mixtures of additional
data and TRANSLATEPROX
(M) or TRANSLATEGOOALRROO
data. Each mixture doubles the agent's s learning data.
Solid lines denote averages over seeds Panda tasks and seeds Goal2D, and shaded regions denote 95% confidence intervals. The upper legend refers to Panda tasks results. We use update ratio a = 1. Panda tasks use the same hyperapameters listed Table 2.
E GENERALIZATION EXPERIMENTS
In this section, we investigate how state-action coverage, reward density, and the augmentated replay ratio affect an agent's generalization ability. For Push, Slide, and PickAndPlace, we train agents over one quadrant of the goal distribution and evaluate agents over the full distribution. For Flip, An agent that generalizes well will achieve a high success rate over the full goal distribution. In general, our observations regarding data efficiency in the main body of this work also apply to generalization.
E. STATE-ACTION COVERAGE
State-action coverage results are shown in Fig. 12. An increase in state-action coverage via augmen- tation increases generalization. In the Panda tasks, using 50% TRANSLATEGOALPRXXIAALL data yields similar performance compared to increased using 50% TRANSLATEGOAL data, indicating that coverage alone can largely explain the generalization improvements with TRANSLATEGOAL. In Goal2D, increased coverage yields better generalization, though considerable gap nevertheless exists between TTRANSI.NTEPROXIIA role in GoalDD. We further investigate this point in the following section.
and TRANSLATE.. Thus, reward density must play larger
E.2 REWARD DENSITY
Reward density results are shown in Fig. 13. In Goal2D, relatively small increase in reward density dramatically improves generalization: TRANSLATERROXIMALOO is roughly on-par with using twice as much peieey-generated data, while TRANSLATEPROXIA (0.00 (0.05) outperforms agents with x8 as much seiieyegeeertedd data. In the Panda tasks, increasing reward density has little effect on data generalization. Just
E.3 AUGMENTED REPLAY RATIO
Augmented replay ratio results are shown in Fig. 14. In Goal2D with TRANSLATE and both Panda tasks with TRENSIA reducing the augmented replay ratio 3 improves generalization performance at convergence. ROTATE achieves 100% success for all values B
20

---

Published as a conference paper at ICLR 2024
Goal2D-v0
PandaPush-v3
PandaSlide-v3
1.0
I0M Success Rale
0.4
0.4
0.4
day
0.2
IQM
0.2
0.2
0.0
0.0
0.0
Updates
Updates
Updates
(a)
(h)
(c)
Figure 13: Learning with WWWNLLATEGOALROOOIIINA and TRANSLATEPROXIMAL())) for various settings of p. Solid lines denote averages 10 seeds in Panda tasks and 50 seeds in Goal2D, and shaded regions denote 95% confidence intervals. We use an update ratio of a = 1. Panda tasks the same hyperapameters listed in Table 2.
Goal2D-v0
Goal2D-v0
1.0
IQM Success Rate
IQM Success Rate
0.8
0.8
0.6
0.6
policy dats
policy policy policy dila
policy
0.4
policy
0.4
Transkie,
Reons 0.
0.2
Trenkben 0.2
025
0.125
00122
Tranokes d 0.2625
Ream. 0 0,0625
0.0
0.0
0.0
0.5
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
255
Updates
Updates
(a)
(b)
PandaFlip-v8
PandaPush-v4
PandaSllid--V
1.0
1.0
Success Rate
Success Rale
0.6
0.6
0.4
0.4
0.4
MQM
5-133
0.2
0.0
3
0.2
0.4
0.6
0.8
1.0
0.0
(c)
(d)
(e)
Figure 14: Decreasing the augmented replay ratio with various augmentations. Solid lines denote averages over 10 seeds in Panda tasks and 50 seeds in Goal2D, and shaded regions denote 95% confidence intervals. We use an update ratio of a 1 for Goal2D and 2 for Panda tasks. Panda tasks use the same hyperapameters used to in Fig. 6.
21

---

Published as a conference paper at ICLR 2024
Environment
Modifications
InvertePPendulum,
cpo from_ to changed from
Lnverte@DoubblcPenduuum
(0, 0, 0, 0.001, 0, 0.6) (0,0, 0, 0, 0.6).
Foot_lefe _emm friction changed from 1.9 to 0.9 so that it matches the right_hipp ______ip_y armature changed from 0.0080 to 0.010 so that it matches left_hip armature. Specify ri right_knee stiffness of so that it matches left _neee stiffness.
Walker2d
geom friction.
Humanoid
Change solver from PGS Newton.
Table 1: Modifications to MuloCo environments ensure intuitive symmetries. We include modi- fications to InvertedPendulum tasks even though we do not consider them in our experiments. We leave them here reference for future work in data augmentation that may require symmetric InvertedPendulum dynamics.
F MuJoCo EXPERIMENTS
In this appendix, we include additional experiments on the following dense reward, continuous state and action MuJoCo environments: Swimmer-v4, Walker2d-v4, Ant-v4, and Humanoid-ve These experiments focus on the augmented replay ratio; the state-action coverage and reward density analysis provided in the main paper is tailored toward sparse reward tasks. With dense reward tasks, we may need consider the full distribution of rewards in the replay buffer, not just the average.
ENVIRONMENT MODIFICATIONS
Some of the common MuloCo environments do not exhibit symmetries that should exist intuitively (e.g. reflection symmetry, gait symmetry, etc.). We found two causes:
1. Asymmetric physics are explicitly hard-coded in the robot descriptor files. We believe these to be typos, and simply update values such that intuitive symmetries exist.
2. Symmetry-breaking numerical optimization algorithms are sometimes used to compute constraint forces and constrained accelerations. particular, some environments use the Projected Gauss-Seidel (PGS) algorithm which performs sequential updates and therefore breaks symmetries where physics should be symmetric. To address this issue, we use Newton's 's method, which performs parallel updates and therefore preserves symmetric physics. We note that while Newton's method MuJoCo's 's default algorithm, some robot descriptor files originally specify the use of PGS.
Table 1 describes all modifications made to environments to ensure intuitive symmetry. To simplify the creation of augmentation functions, we additionally exclude constraint forces and center-of-mass quantities from the agent's observations, since their interpretations are not well-documented and difficult ascertain.
22

---

Published as a conference paper at ICLR 2024
(a) Ant-v4: ROTATE
(b) Ant-v4: REFLECT
(c) Humanoid-v4: ROTATE
Figure 15: Visualizations some augmentations in MuJoCo environments.
F.2 DATA AUGMENTATION FUNCTIONS
We consider the following dynamics-invariant augmentations:
Swimmer-v4
REFLECT:
Reflect joint angles and velocities about the agent's central axis. The reward is unchanged.
Walker2d-v4
Swap the observations dimensions the left and right legs. The reward unchanged.
REFLECT:
Ant-v4
REFLECT:
Swap the observations dimensions of the front and back legs. Unlike the other reflections, this augmentation affects the reward. In partieular, if the observed transition moves forward with velocity u. the reflected transition will move backwards with velocity v. Thus, this reflection flips the sign of the "forward progress" reward term in the reward function.
ROTATE: Rotate the agent's orientation by sampled uniformly random from |-m/6,
Humanoid-v4
Swap the observations dimensions of the left and right arms/legs, and reflect torso joint angles, velocities, and orientation about the agent's 's central axis. The reward unchanged.
REFLECT:
ROTATE: Rotate the agent's orientation by sampled uniformly at random from 1-n/3, r/3).
23

---

Published as a conference paper at ICLR 2024
x1 policy data
....
x1.25 policy data
Reflect, , 1.0
Reflect. f = U.S
Reflect. B 0.5
Sutmmer-u4
Walker2d-44
Antov4
Humanzid-o4
5000
5000
4000
4000
Ret.m
3000
2000
2000
16033
1500
1000
0.7
0.6
06
1.0
0.2
0.0
1.0
00
06
Updates
Updates
Updates
Updates
Figure 16: Lowering the augmented replay ratio for the REFLECT augmentation. Solid lines denote averages over seeds, and shaded regions denote 95% confidence intervals.
policy
policy
Rolate,
Rolale,
Rolate, B
Ant-v4
Humanoid-v4
6000
5000
5000
4000
Return
Retum
3000
2000
1000
1000
0
Updates
166
Updates
1c6
Figure 17: Lowering the augmented replay ratio for the TRANSLATE and ROTATE augmentations. Solid lines denote averages over seeds, and shaded regions denote 95% confidence intervals.
F.3 AUGMENTED REPLAY RATIO
We repeat the same augmented replay ratio experiments detailed in Section 5.1.4 for MuJoCo tasks. We decrease the replay ratio by generating more augmentations per observed transition while keeping the amount of augmented data used in policy/value function updates fixed, i.e., we increase the augmentation ratio m while fixing the update ratio a. We consider all augmentations listed in Appendix F.2.
For the ROTATE augmentation, we fix the update ratio at a = and generate m = 1,, 1,2, 4 augmented transitions per observed transition, corresponding to augmented replay ratios 3 = 0.5, 0.25, respectively. Even though REFLECT can only generate a single augmented transition per observed transition,
nevertheless study the augmented replay ratio as follows. Rather than generating an augmenting every observed transition, we instead augment transitions with some probability p such that the augmented replay ratio is 1/p in expectation. We can think of p fractional augmentation ratio. We fix the update ratio at a = 0.25 and consider p = 0.25, 0.5, 1, again corresponding to augmented replay ratios 3 1, 0.5, 0.25, respectively.
Results are shown in Fig. 16 and 17. A lower augmented replay ratio with REFLECT yields slight improvements in data efficiency for all environments except Humanoid-b4, where performance is largely unchanged. We observe much larger improvements with lower augmented replay ratio in our core sparse reward tasks (Section 5.4.
24

---

Published as a conference paper at ICLR 2024
Episode length
most 50 timesteps
Evaluation frequency
10,000 timesteps
Number of evaluation episodes
80
Number of environment interactions
600K (Push), 1M (Slide, Flip), 1.5M (PickAndPlace)
Random action probability
0.3
Gaussian action noise scale
0.2
# of random actions before learning
1000
Observed replay buffer size (default)
10
Augmented replay buffer size (default)
1 10
Batch size (default)
256 (Push, Slide), 512 (Flip, PickAndPlace)
Update frequency
Every timesteps (observed replay ratio of 0.5)
Network
Multi-layer perceptron with hidden layers (256, 256, 256)
Optimizer
Adam (Kingma and Ba, 2014)
Learning rate
0.001
Polyak averaging coefficient (r)
0.95
Table Default hyperparameters used in all Panda tasks.
G TRAINING DETAILS
We use the Stable Baselines3 (Raffin et al., 2021) implementation of DDPG (Lillicrap et al., 2015) and TD3 (Fujimoto et al., 2018) with modifications incorporate augmentation into the RL training loop. In Panda tasks, we use DDPG since we found that it performs substantially better than TD3, This observation was also made by Gallouédec al. (2021). All Panda experiments use the default hyperparameters presented in Table 2. These parameters are nearly identical to the those used by Plappert al. (2018) and Gallouédec al. (2021). The augmented replay ratio experiments in Fig. 6 and update ratio experiments in Fig. D.1 use different values for two hyperparameters specified below:
Random action probability: O
Update frequency: Every timestep (observed replay ratio of 1)
We ran all experiments on a compute cluster using a mix of CPU-only and GPU jobs. This cluster contains mix of Tesla P100-PCIE, GeForce RTX 2080 Ti, and A100-SXM4 GPUs. Due limited GPU access, we only used Us for the augmented replay ratio experiments in Section 5.1.4, since these were our most computationally demanding experiments. We ran state-coverage, reward density, and generalization experiments on CPU only. CPU jobs took 12-36 hours each depending on the training budget, and GPU jobs took up to 16 hours each.
25


Reviews:
[{"review_id": "ELq46s0lvQ", "reviewer": "Reviewer_ynPP", "summary": "This paper analyzes the effects of data augmentation in reinforcement learning. A new evaluation framework is built, and three important indexes of data-augmented RL are studied empirically.", "questions": "Could you explain Definition 3 and its purpose for the following analysis? An example would be helpful.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The topic is interesting in providing a study on the effectiveness of data augmentation in reinforcement learning. \n\nThe paper presentation is clear, especially since the problem model and proposed framework are easy to follow.", "weaknesses": "There are many data augmentation ways, and the paper has also listed many of them. However, in experiments, only 2-3 basic ways are used. Considering the four typical testing environments, the conclusion of empirical analysis is quite limited.\n\nThe core idea of this paper can be very helpful for data-inefficient RL that needs augmentation, but it lacks theoretical analysis to support the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hqfdztLYAa", "reviewer": "Reviewer_UZG5", "summary": "The paper studies the effective use of the dynamics-invariant data augmentation (DA) in reinforcement learning (RL). The authors propose an empirical framework to investigate the effectiveness of DA in RL, where they particularly focus on the impact of the augmented replay ratio, governing the frequency of using augmentations. Based on this, they identify the ratio as one of critical hyperparameters in RL, and demonstrate that it is sometimes better to set the replay ratio lower, whereas, intuitively, it seems always better to use higher ratio as a part \nof maximizing the benefit from the augmentation. Besides this, the authors provide a set of insights on using DA in RL, in particular, when reward is sparse.", "questions": "It would be great if additional experiments, further discussion, or navigation to what I’ve missed can be provided to address the comments and questions in the weakness and what follows.\n\n- Do the findings hold if DA is used for more than just augmenting data, e.g., policy distillation or representation learning? The other uses of DA may extract the full potential of DA in RL, and show somewhat different observations.\n\n- As we all know, choosing hyperparameters is critical in any machine learning. Hence, I want to ensure that every hyperparameter has been optimized for each scenario.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The paper identifies a critical hyperparameter, the augmented relay ratio, in RL, which may overlooked before.\n\nThe authors propose a framework to empirically study the impact of data augmentation in RL.\n\nThe paper is well-written and easy to follow.", "weaknesses": "My major concerns are the weak messages of the empirical findings. The authors provide some useful (yet somewhat intuitive) insights on using data augmentation in RL, but no effective usages in practice. It would be better to clarify the use of their findings. In my understanding, the most useful message in practice is to consider tuning the augmented reply ratio. It would be helpful to provide practical strategies to use DA effectively, based on the empirical findings.\n\nThere has been a line of works to study the impact of data augmentation in RL, and maximize its efficacy in RL, e.g., [A,B,C]. It has been observed that maximally using the data augmentation in RL training is sometimes worse than not using the augmentation. In some sense, this coincides with this paper's major finding (low augmented replay ratio is better), although they focused on the vision-based RL, that is different from the one considered in this paper. It is necessary to discuss differences and coincidences between this work and existing ones.\n\nIn addition, the proposed framework to study the effectiveness of data augmentation seems straightforwardly designed. It would be helpful to clarify the technical novelty of the framework.\n\n[A] Laskin, Misha, et al. \"Reinforcement learning with augmented data.\" Advances in neural information processing systems 33 (2020): 19884-19895.\n[B] Raileanu, Roberta, et al. \"Automatic data augmentation for generalization in deep reinforcement learning.\" arXiv preprint arXiv:2006.12862 (2020).\n[C] Ko, Byungchan, and Jungseul Ok. \"Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning.\" Advances in Neural Information Processing Systems 35 (2022): 33289-33301.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iw1rKahfmk", "reviewer": "Reviewer_Y7KF", "summary": "This paper is an empirical study of data augmentation in reinforcement learning, which has been shown to improve results. The authors hypothesize three possible causes for the improvement, 1) state-action coverage, 2) reward density, and 3) augmented replay ratio. For the off-policy and sparse-reward setting, a study is designed that disentangles these three causes as much as possible, on environments from panda-gym and 2D navigation. The results indicate that increasing state-action coverage and decreasing augmented replay ratio are more important, while increasing reward density is less important, although specific conclusions are task-dependent.", "questions": "1. How were the algorithms used in the study chosen?\n2. Is there intuition about how the differences in the tasks affects the results? For example, in figure 3 the \"x2\" and \"x4\" results for policy data and augmented data are similar, but this is not true in figure 1.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "# Originality and significance #\n\nAs far as I know, this is the first systematic study on the underlying causes of the improvements from data augmentation in RL. As such, I think the results would be helpful for researchers designing similar algorithms in the future.\n\n# Clarity #\n\nThe paper is written very clearly and the approaches are straightforward to understand. The separation of results into the main text and supplement is reasonable.\n\n# Quality #\n\nThe approach used to disentangle the three hypothesized causes are clever. Uncertainties are also presented in the graphs.", "weaknesses": "The authors state many of the weaknesses of the work in the last paragraph of the paper: limited range of environments, restricted data augmentation framework, and incomplete set of properties investigated. One that they did not state was the limited range of algorithms. However, I think aside from the limited range of environments, those weaknesses are a reasonable consequence of the approach used to disentangle the three causes. Addressing them would make the disentanglement much harder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0t29LKnoEH", "reviewer": "Reviewer_Mi19", "summary": "This paper studies the effects of data augmentation in off-policy RL, notably the number of augmented transitions per observed transition, the number of augmentations per update, and the ratio of real data vs augmented data used in updates. The intermediate effects studied are the resulting diversity on state-actions and the diversity of rewards. They find that the state-action coverage is the most important outcome of data augmentation that leads to success, and this effect can provide equal (or even better) performance than drawing more real (diverse) data from the environment.", "questions": "My primary question is wrt f, e.g., how is this guaranteed that f generates valid transitions, what are the consequences of f not generating valid transitions, what are the properties of f in the experiments provided, etc. Could you please clarify these points in the experimentation section as well as clarify earlier in the work whether or not f is chosen to have the characteristics noted (validity, etc). But how much does this matter? Is it ok to have some invalid transitions in the augmented data, how might these change the results, etc?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well presented and clear: I wasn't able to find any flaws in the story nor the arguments. Motivation is very clear, and they do an excellent job isolating various factors in augmentation-based (off-policy) algorithms and provide clear experimentation. The experiments seem insightful and I believe that the conclusions are sound.", "weaknesses": "Very little weaknesses, though I think mostly I would like to see a little more clarity wrt ensuring f generates valid transitions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)