Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Control
Authors: Jinzhu Luo
Dingyang Chen
Qi Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement learning
Data augmentation
Continuous control
Euclidean symmetry
Primary area: reinforcement_learning

Abstract: Data augmentation creates new data points by transforming the original ones for an reinforcement learning (RL) agent to learn from, which has been shown to be effective for the objective of improving data efficiency of RL for continuous control. Prior work towards this objective has been largely restricted to perturbation-based data augmentation where new data points are created by perturbing the original ones,
which has been impressively effective for tasks where the RL agent observe control states as images with perturbations including random cropping, shifting, etc. This work focuses on state-based control, where the RL agent can directly observe raw kinematic and task features, and considers an alternative data augmentation applied to these features based on Euclidean symmetries under transformations like rotations. We show that the default state features used in exiting benchmark tasks that are based on joint configurations are not amenable to Euclidean transformations. We therefore advocate using state features based on configurations of the limbs (i.e., rigid bodies connected by joints) that instead provides rich augmented data under Euclidean transformations. With minimal hyperparameter tuning, we show this new Euclidean data augmentation strategy significantly improve both data efficiency and asymptotic performance of RL on a wide range of continuous control tasks.

Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Control
Jinzhu Luo University of South Carolina j jinzhu@email. sc. edu
Dingyang Chen University of South Carolina dingyangdemail sc. edu
Qi Zhang University of South Carolina qz50cse sc
Abstract
Data augmentation creates new data points by transforming the original ones for reinforcement learning (RL) agent to learn from, which has been shown to be effective for the objective of improving the data efficiency RL for continuous control. Prior work towards this objective has been largely restricted to perturbation- based data augmentation where new data points are created by perturbing the original ones, which has been impressively effective for tasks where the RL agent observes control states as images with perturbations including random cropping, shifting, etc. This work focuses state-based control, where the RI. agent directly observe raw kinematic and task features, and considers an alternative data augmentation applied to these features based on Euclidean symmetries under transformations like rotations. We show that the default state features used in exiting benchmark tasks that based on joint configurations to Euclidean transformations. We therefore advocate using state features based on configurations of the limbs (i.e., the rigid bodies connected by the joints) that instead provide rich augmented data under Euclidean transformations. With minimal hyperparameter tuning, we show this new Euclidean data augmentation strategy significantly improves both data efficiency and asymptotic performance of RI. on wide range of continuous control tasks. Our code available on GitHub!.
not amenable
Introduction
While reinforcement learning (RL) has enjoyed impressive success on continuous control problems, especially when empowered by expressive function approximators like deep neural networks, im- proving its notoriously poor data efficiency remains challenging. Recently, exploiting the idea of data augmentation, the RL community has made significant progress on improving data efficiency as well as the asymptotic performance of RL for continuous control with the agent observing images as the state representations [1. 2. 3. 4]. CURL [1] utilized data augmentation for learning contrastive representations [5, 6] out of their image encoder, as any auxiliary learning objective in the RL setting. Other works, such DrQ [3, 4] and RAD [2], directly used image data augmentations for RL without any auxiliary objectives. Despite these technical differences how data augmentation integrated into RL, these works share the key procedure of perturbation- based image augmentation: create new data for learning, the image in a state representation is transformed by applying perturbations such random crop, random shift, color jitter, etc.
Instead of image-based continuous control, this paper focuses on the more primitive problem of state- based continuous control, where the agent can observe raw physical quantities such position and velocity as the features to represent the underlying world state. Like the image-based case, existing work on state-based data augmentation has been relying introducing perturbations to original state
ltttps://github.co www.//o///nnnnnaceeee httaa///tnauuao/////eeeeellllll Eddand
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

features, examples including adding Gaussian noise and scaling the features with random amplitudes [2]. However, unlike the image-based case, the perturbed data for state-based continuous control only leads to limited improvements, sometimes even negative effects, on the data efficiency of RI., as found in existing work [2] and our experiments in Section Such ineffectiveness due to the fundamental difference between magge-based and state- state-based features: from a control-theoretic perspective, the perturbed physical quantities as state-based features are largely uncorrelated with the original ones, in the sense that although the original transition comes from the ground truth dynamics and reward functions, the perturbed transition does not necessarily; this is in contrastive difference from images, where perturbations such as shifting create new views of the same underlying world state.
To overcome the limitations of perturbation- based data augmentation, this paper advocates the alternative of Euclidean data augmentation for state-based continuous control. The key idea is that, because our control tasks operate in the 2D/3D space, they exhibit Fuclidean symmetries: transformations such as rotation and translation leave the transition dynamics and reward function invariant. Due to the invariancy, these symmetry-pene preeeeving Euclidean transformations are ideal operations for data augmentation, because the transformed data will be guaranteed to be valid samples from the task's s dynamics and reward.
While there exists prior work on Euclidean symmetries in RL [7, 8, 9, 10], the idea of exploiting them as a data augmentation method for state-based continuous control is surprisingly underexplored. This is because current benchmarks [11, 12] by default use the joint configurations of the robot- like agent as its state features, which are mainly angular quantities that are invariant to Euclidean transformations and therefore not amenable to data augmentation, as we will explain in detail in Section 4. To overcome this, we innovatively use configurations of the limbs, i.e., the rigid body links connected by the joints, as an equivalent state representation to replace the default joint-based one. Because limb configurations are specified by physical quantities such as linear position and velocity, they are equivariant under Euclidean transformations and therefore provide rich augmented data.
Our algorithmic innovations based on the introduced ideas lead to significant improvement in data efficiency and asymptotic performance of RL for state- based continuous control. Building from DDPG [13] as the base RL algorithm, our limb-based state representation alone improves the performance on most tasks from the DeepMind Control Suite [12]. a standard continuous control benchmark for RI., and additional Fuclidean data augmentation
necessary to obtain the best performance for almost all tasks, especially for the ones that have large degrees of freedom and are historically hard to solve. To name a few hardest tasks, on the Humanoid run task, standard DDPG achieves an episode reward below 100, while our method achieves 150, both after 5M timesteps; on Hopper3D hop, standard DDPG achieves an episode reward below 40, while our method achieves more than 200, both after 2M timesteps.
2 Related work
Data augmentation for image-based continuous control. Existing works in data augmentation for RL mostly have focused on the online setting for image-based continuous control tasks. This is because it relatively straightforward to obtain augmented images (e.g., through random crop- ping/shifting),, which enables learning representations of the high-dimensiona image input that facilitates optimizing the task reward. In this line of work, CURL [1] utilizes contrastive representa- tion learning to the image representations, jointly with the RL objective of reward optimization; RAD [2] and DrQ [3, 4] use the augmented data directly for RL without any auxiliary objective. SVEA [14) focuses on improving stability and sample efficiency under image-based data augmentation by mitigating the high-variance Q-targets. ColT [15] works on maintaining the invariant information unchanged under masee data augmentation. Different tasks often benefit from different types of data augmentation, but the manual selection is not scalable. The work of [16] automatically applics the types of data augmentations that are most suitable for the specific tasks.
Data augmentation for saste continuous control. This paper focuses on data augmentation for stte-based continuous control, which only a few prior works have explored. Specifically, RAD explores the augmentation of injecting noise to state variables through additive Gaussian or random amplitude scaling [2]. Different from these unprincipled, perturbation-baser augmentation, this paper advocates a principled augmentation method through Fuclidean symmetries for state-based continuous control. Corrado et al. [17, 18] also consider principled data augmentation transformations that are not

---

perturbation based but focus on better leveraging existing transformations, drawing their conclusions mostly from robot navigation and manipulation tasks, while we propose novel transformation for robot locomotion. Pitis et al. [19, 20] propose data augmentation transformation that requires local (causal) independence, so that augmentation can be performed via stitching independent trajectories from decomposed, independent parts, which is useful for tasks such as particles moving and two-arm robots with a static base. We instead focus on locomotion tasks that do not exhibit sparse kinematic interactions between limbs and therefore cannot benefit much from their method. For example, the legs of a cheetah-like robot are connected through the moveable torso, which we cannot decompose and stitch their separate trajectories.
Euclidean symmetries in RL. Symmetries in RL refer to the structural property of invariant transition dynamics and reward functions under certain transformations, leading to the existence of invariant optimal values and policies [7. 8]. Euclidean symmetries thus correspond to the transformations being translations, rotations, and/or reflections, which are applicable when the state/action space Euclidean. Most prior works focus on image-based control problems, e.g., Atari games and robot manipulation from image observations [9]. or gridworld-like domains exhibiting only discrete symmetries (e.g., 4-way Cartesian rotation) [8, 21J. A closely related work by Abdolhosseini et al. [22] uses reflection-based data augmentation (they call them "mirror") for locomotion. They perform trajectory-lewe transformations for on on-policy RL, which is technically not sound and does not yield significant improvement over the no-augmentation baseline (see Figure 3 therein). Instead of relying on data augmentation, some of these works employ equivariant neural networks as their agent architectures to exploit the symmetries [8. 9]. which often incurs significantly higher computational cost. The reason that existing works largely overlook data augmentation based on Euclidean symmetries in state-based continuous control is two-fold: 1) most standard benchmark tasks operate in the 2D space [11, 12], and these planar tasks exhibit limited Euclidean symmetries compared to the 3D counterparts; and 2) the benchmark tasks employ the state representations based on the joints' configurations and velocities, which in nature exhibit very limited Euclidean symmetries. This paper addresses these issues unlock the potential of data augmentation through Euclidean symmetries for state-based control via RL.
3 Preliminaries
3.1 Online RL for state-based continuous control
We formulate a state-based continuous control task (or, simply, task) as a Markov decision process (MDP), defined as a tuple (S.A. r,y)) 7). At each discrete time step t, the agent fully observes the state in the continuous state space, 8; E S C Rd. and chooses an action in the continuous action space to take, a, E A C R"; the next state is sampled from the transition dynamics p with conditional probability density p(s++1 118,, at), while the reward function r S A + R yields a scalar feedback r; r(st, az). The agent uses policy π : S - A(A) to choose actions where (AA) is the collection of probability measures on A, yielding a trajectory of states, actions, and rewards, (so, do, ru, .). where the initial state so is sample from initial state distribution with density do and a ~ x--(s)) Without prior knowledge of p and r, the reinforcement learning (RL) agent's 's goal is to obtain a policy from its experience (i.c., yielded trajectories) which maximizes the expected cumulative discounted reward, E IEer Y'r,] where y E [0, 1). The online RL setting assumes the agent has the ability interact through the MDP to obtain an increasing amount of trajectories while refining its policy.
3.2 Deep Deterministic Policy Gradient
Deep Deterministic Policy Gradient (DDPG) [13] is a state-of-the-at online RL algorithm for continuous control, which parameterizes and learns a critic Qe : S x A- policy मे : S - A. The critic trained by minimizing the one-step temporal difference (TD) error L())= = (.......r 1- [[Qe (St, (
+ IR and deterministic
{)) where Dontine is the replay buffer storing past trajectories and 0 an exponential moving average of the critic parameter, referred to as the target critic. Concurrently, the policy is trained by employing Deterministic Policy Gradient (DPG) [23] and minimizing L(Φ) = RRg, [-Qo (8e, ne (st))] so that T (8:) approximates arg max, Qe (s;, a). The policy, after adding noise for exploration, is used as data-collection policy
3

---

to generate and store new trajectories in replay buffer D Donline, which is interleaved with the critic and the actor parameter update steps.
3.3 Perturbation- and symmetry-baser data augmentation in RL
Perturbation-basee based data augmentation. Data augmentation generates new, artificial data through transformations of original data. Better data efficiency can often be achieved by learning from both the original and augmented data. For mastering continuous control tasks via RL, prior works perform data augmentation by perturbation, through 1) transformations of adding noise to original data, examples including performing random image cropping, color jittering, translations, and rotations when states can only be observed and represented as images (i.e., image-based continuous control) [24, 25, 1, 2, 3, 4] and 2) transformations of adding Gaussian noise and random amplitude scaling when states (e.g.. positions and velocities) can be directly observed by the agent [2].
Symmetry-basee data augmentation. This work focuses on state-based continuous control and departs from prior works by performing data augmentation through MDP symmetries [7, 8]. Symme- tries in an MDP, if existing, can described by a set of transformations on the state-action space indexed by g E G. where such transformation leaves the transition dynamics and reward function invariant. Formally, we define a state transformation and a state-dependent action transformation as TS S + S and T'A : A + A, respectively, and the invariance is expressed accordingly as, for all E G, E. A.
p( = p (T, [5]] 19 [s), Ty. and ar - (Ty (I
(I)
The invariance naturally makes these transformations sound and effective for data augmentation pur- poses, especially in continuous control tasks where transition dynamics is often (near- Xdtterministic. Next, we detail our data augmentation method based on Euclidean symmetries for continuous control.
4
Our method: Euclidean data augmentation for continuous control
4.1 Kinematics of our continuous control tasks
We consider the continuous control problem for an agent (e.g., a robot) consisting of n rigid bodies, referred to as limbs, that are connected by joints to form certain morphology and can move in the 2D planar space or the 3D space. Examples of such agents include 2D Cheetah and 2D Hopper from DeepMind Control Suite (DMControl) [12] well as their 3D variants [26], which will be presented in our experiments. Without loss of generality, we present our method in the 3D setting.
Tree-structured morphology and actuation. The connected limbs form a certain morphology that can be represented as an undirected graph, where each node i E {1..... n) represents a limb and an undirected edge (i, exists if there is a joint connecting nodes/limbs i and j. We focus on the case where the graph is connected and acyelic to become tree, where the root node, indexed i = 1, is often the torso or the base in the agent's 's morphology. Any two connected limbs are rigidly attached to each other via their joint to provide degrees of freedom (DoFs) between the two limbs. A joint provides either 1-, 2-, or 3-DoF rotation for the child node/limb relative to its parent: in general, as a 3D rigid body, the child can rotate about yaw-, pitch-, and/or roll-axes relative its parent, creating 3 DoFs; for a 2D planer task, the agent operates in the zzapl and the joints therein can only rotate about the y-axis (pitch), creating only DoF. The joints are actuated produce scalar torque for each of these rotation DoFs. We let d, E 21, 3} denote the number of DoFs of the joint with node i > as its child, and the collection of d, scalar torques is denote as a, E Rd resulting in the action for the agent as a (a,)i>1 E RMT where m Ex>1 d, The root node, as a 3D rigid body, has up to 6 DoF's relative to the global frame, and we let d1 S 6 denote the number of its DoFs.
Kinematic features. As the action of torques specifies acceleration, a Markovian state of such an agent has to specify its configuration, i.e., how the limbs occupy the space and how the joints are rotated, as well as its velocity, i.e., the time derivative of its configuration. We refer to these physical quantities as the agent's kinematic features and enumerate here several of them that will be closely related to our discussion:

---

global frame
For the joint with body i as child: q 4: E joint position and velocity Fla Fid, E ; rotation axes
torso
back thigh
front thigh
For body i, relative to the global frame: P. Vj R position and linear velocity R; E Rax3 orientation
back shin
front shin
back foot
front foot
i angular velocity
Figure I: Illustration of Cheetah from DMControl, including its rendering, tree-structured morphol- ogy with the nodes being the limbs and the edges the joints, and the state features.
Oi q: E Rdd respectively denote the configuration and the velocity (i.e., the configuration's 's time derivative) of the joint with limb as child. For i> 1, we have d; S corresponding to each active joint rotation axis, and consists of the rotation angles. For i 1, there is no joint with limb = 1 (i.e., torso) its child; in this case, Qı Rdd specifies the configuration corresponding to all di < DoF' the limb.
1 ... Fi.d, E IR3 denote the d; rotation axes of the joint with limb > 1 as its child, relative to the global frame.
p., U, E I² respectively denote the (translational) position and linear velocity (i.c., the position's time derivative) of limb i 1, relative the global frame.
R, 3x3 and W; R respectively denote the Cartesian orientation and angular velocity (i.e., the orientation's 's time derivative) of limb 2 1, relative to the global frame.
Task features. Our continuous control tasks are mostly of locomotion, i.e., the agent is tasked to move (part of) itself from one place to another. In tasks where the agent is tasked to move in certain direction (e.g., Chestah), we let & IR denote the unit vector in that direction. In tasks where the agent aims to reach target point p.. E IR33 with one of its limb i (e.g., Reacher where = 3 being the last limb), we let = p. P. R denote the vector from the limb to the target point.
Example: 2D/3D Cheetah. The original Cheetah from DMControl operates in the æz-plane [12] with the target direction being f= [1 0 0). i.e., the agent is tasked to move along the positive r-axis. Its torso the root node and has dj
DoFs: moving along the z-axis with position p E R. moving along the z a-axis with position p E IR, and rotating about the y-axis with rotation angle 3y E [o, 2.T). and correspondingly we have
q1 = (pT. pi. (T) E R The other limbs (front/back thigh, shin, and foot) form the front and back legs attached to the torso, forming the morphology tree as shown in Figure For the 2D planar case, the joints can only rotate about the y-axis, and therefore, for i E (2.... 7}. have d; L, q: = (33) e R with 8.e being the rotation angle, and its only rotation axis Ft, is fixed along the y-axis. our experiments, we extend this 2D Cheetah its 3D variant, where the torso has all d1 DoFs to freely move the in 3D space and each of the 6 joints, E {2. 7), has d, 3 DoFs be able freely rotate about the yaw-, pitch-, and roll- axes (F Fi,,, Ft.a).
R₂ E 50g
4.2 SO (3)- augmentation
If the agent is placed in the free space (free of obstacles and external forces), its transition dynamies and reward function are invariant to the transformations including translations, rotations, and reflections. Formally, these transforma- tions together form the Fuclidean group G = Fx()) in Equation (1), where the state should include the agent's kinematic features (i.c.. configuration and
Figure 2: SOg(3)
rotation in Cheetah.
5

---

velocity) and the task features. This is the basis of our method of Euclidean
data augmentation.
For the tasks in our experiments, the agent is not placed in the free space and, instead, is placed on the level surface of æy-plane (i.e., the ground) and subject to the external force of gravity g E IR'S towards the negative -axis. Therefore, the task is not invariant arbitrary rotations/reflections but only to rotations about the direction of ğ (or equivalently, the z-axis), which subsumes reflections along this direction. For the reasons mentioned above, our Euclidean data augmentation only includes rotations about y. which form a subgroup of SO(3) (which includes all 3D rotations) and is denoted with its matrix representation as S0g(3) = {R& E R3x3 R R = I, det R = Rg = g). Alternatively, the matrices can be parameterized by the angle rotated about z-axis (i.e., yaw) as [0, 2n) as (3g(3) (R. R a E [0, 2T), B 7 0)} where a, B, and y are yaw, pitch, and roll angles, respectively. We will abbreviate the matrix as R. since 3 = 7
Note that other Euclidean transformations, i.c., translations reflections, are not available in our tasks: As explained later, the agents in the tasks use egocentric representation and therefore it's already munsslationinvariann Reflections require gait symmetry (c.g.. the left leg has the same length, stiffness, etc.), which the agents in the tasks do not necessarily have. Some prior work has modified the agent enforce gait symmetry before applying reflections (e.g., Corrado and Hanna [18)), but we do not.
For state rotated by R., the kinematic features and task features are accordingly transformed in the following three ways (see Figure for an illustration in Cheetah):
Equivariant features: We have denoted these features with arrow as 3D vector ã E R which will be rotated into R, E IRS, including ,,,,,... Fi,d, P., Uú columns of R. W., and E,
Invariant features: For the joint with a non-torso limb i > as its child, its configuration q: specifies the rotation angles to the active rotation axes. Therefore, {q:: 4:he>1 and torque action a will stay invariant.
Transformation of qı and श:: Note that elements of the torso's position कi are also part of qı- This part of qı will be transformed accordingly. The rest of qı includes 3i, By and/or 3r, the yaw, pitch, and/or roll angles of the torso. Since rotation Ra is a yaw-only rotation, we have 3i + of + a while o and ST will stay invariant. Similar arguments apply to ğ1.
We below present a few implementation details S0g(3)-data augmentation in our experiments.
L Limb-based kinematic state representation. Besides the task features, the Markovian state should include, for the agent itself, kinematics features that are sufficient to represent its configuration and velocity. Not all kinematic features introduced in Section 4.1 are independent and therefore sufficient kinematic state representation needs only to include a subset of them. Current benchmarks for RL-based continuous control (e.g., DMControl [12] and Gym- Mym-MuJoCo [11]) all employ the joint- based kinematic state representation, which includes only joint configurations and velocities {q:, q h2,, which is indeed valid kinematic state representation because all limbs are rigid bodies. However, as we have discussed, this joint-based kinematic state representation is not amenable to (0g(3)-date augmentation, since most of its features are invariant (only qi and ý not invariant).
To provide rich augmented data under (3), we use an alternative kinematic state representation that is mostly based on limb features, including R,, w1, {p., ü U.h2 and (Fi.,. Fe,d, h:s For torso its configuration and velocity are included in {Ri, Pi, J.). For limb i > 1, its translational position and velocity are included in (T;, J. its orientation is included in its rotation axes { Fi.d) J. so we do not further include R, and its angular velocity can be recovered from its rotation axes and its velocity, so we do not further include wi
The kinematic features listed above are both sufficient and necessary in general for an agent operating in the 3D space. For 1-DoF joint (d:- = 1) that typical in 2D planar tasks, the orientation of the torso, RT, is sufficient to determine the orientation of its active rotation axis. Therefore, we exclude the rotation axes for all the 1- I-DoF joints and include only for the 2- and 3-DoF joints, resulting in the final set of features in our limb-based kinematic state representation:
R, 11 (p., 8.222. (rr1, Fi.d : 1, 1, > 1}.
(2)

---

Egocentric kinematic features. In these benchmark continuous control tasks, the agent can only observe its egocentric position, that is, it observes the position of its torso as the origin and the positions of other limbs and the target relative to its torso. For example, when using the joint-based, kinematic state representation for 2D Cheetah, the configuration of torso = (Pi, pi, E R. is observed as (0.pi. pi. ). Consistently, our limb-based representation subtracts the torso's translational position for every limb, i.e.. P: + p: P1 for all limbs i 2 1.
Additional sensory observations. Besides the kinematic and task features, the agent has additional sensory observations in its original state representation that can be classified either equivariant or invariant features under S0g(3) rotations. For example, Hopper from DMControl has two sensors for detecting the touch of its two feet, each yielding invariant scalar feature. Refer to Table 2 in Appendix A.1 for a complete list of sensory observations for all tasks. Our complete state representation includes these sensory inputs, along with the kinematic and task features, to ensure equivalence to the original state representation.
4.3 Integration with DDPG
Our data augmentation method can be applied to any off off-policy RL algorithm, and our empirical study applies it to DDPG as follows. After sampling a batch of B transitions of the form (st, dt, Tr, 8t41) from online replay buffer Danlines we randomly choose a subset of Ba B) transitions and perform random rotations R. S3)(3) to them, with randomly chosen for each of the Bang transitions. Baug is the only additionaa hyperparameter that our method introduces on top of the original DDPG, Specifically, transform states s, and st+1 as described in Section 4.2, keep action a, and reward rt invariant, and use the transformed transition to compute the DDPG losses (cf. Section 3.2).
Experiments
Environment and tasks. All the tasks in our experiments are provided by the DeepMind Control Suite (DMControl) [12] powered by the physics simulator of MuJoCo [27], which has become a common benchmark for RL-based continuous control. Specifically, we include 7 tasks originally from DMControl: Cheetah run, Hopper hop, Walker_ run, Quadruped run, Reacher hard, Humanoid run, Humanoid stand, and our modified Cheetah3D run, Hopper3D _hop, and Walker3D run, which are the 3D variants of their original 2D planar counterparts. The original tasks involving Quadruped and Humanoid are already 3D, while it is not straightforward to extend Reacher _aar to 3D variant. This results in total number of 10 tasks. For all tasks in DMControl, an episode corresponds to 1000 steps, where per-step reward is in the unit interval, i.e., Tr E [0, 1]. The original tasks DMControl employ the joint-based kinematic state representation. We use MuloCo's 's built-in data structures and functions (e.g., mj Data. xpos for the translational position, mj mjtaa. object_velocice velocito ty for linear and angular velocity [28]) to obtain the kinematics features needed for our limb-based representation. Refer to Appendix A. I for a detailed description of the state features for all the tasks.
Baselines. We build our data augmentation on top of DDPG, following the implementation in [10] that has achieved of-te--f-th-arn performance on the chosen tasks (comparable or better than, e.g., TD3 [29], Soft Actor-Critic (SAC) [30], e.g., refer [10] for comparison). Refer Appendix A.2 for a full list of the DDPG's hyperparameters. For reference, we also run and provide results of standard SAC. Our method introduces only one additional hyperparameter, Baug. the number of transitions to be transformed in batch of B transitions. We perform a hyperparameter search over Baug/B =: Paug E {0, 25%, 50%, 75%, 100%} separately for each task, while keeping all other hyperparameters the same as DDPG default. We compare our SO (0g(3)-das) (3)-data augmentation with two alternatives considered in the prior work of [2], both operating on the original joint-based kinematic state representation: The Gaussian noise (GN) method adds standard Gaussian random variable to the state vector, i.e., > s + where (0, I); and random amplitude scaling (RAS) multiplies the uniform random variable to the state vector element-wise, i.e., s + C with O denoting element-wiss product, where = ~ Uni[a,,3 with a = 0.5, 3 = 1.0 as chosen in the prior work of [2]. We additionally compare to using equivariant neural networks, specifically SEGNN [31], to instantiate the agent's 's policy and critic, as an alternative method to exploit the Fuclidean symmetries. Refer to Appendix A.3 for the details of our SEGNN implementation.
7

---

Pay 25%
- 75%
600
250
SAC
porr
2 100
400
8
200
DOPE GN
DOPE + RA5
DOPE Duu
100
porG Oum 25%0
100 Time step (=1)))
130
30
Time step Ox
Time step (x
(a) Cheetah_run run
(b) Cheetah3D_rut
(c) Hopper_hop
Dony 50%
800
Dany 52%
Daay 25%
250
700
6:00
1.00
400
D0
100
100
60
200
50
40-
0
50
100
1.5
1.0
1.50
200
50
1.00
1.6
200
Time step 201)
Time step (=10f)
Time step (x20
(d) Hopper3D hop
(c) Walker
Walker3D run
Plop 25%
175
Plup 100%
Plop -
150
125
5.00
1000
2200
103
230
50 10s 150 203 250 3h0 350 1h aso YOu
50 107 155
200 250 301 300 the to w0
Time step 14
Time step 1x
"Tmee step (410%)
(g) Quadruped_
(h) Humanoid_mme
Humanoid_ssare
Figure 3: Learning curves comparing data efficiency of our method again all baselines except for SEGNN on out of the 10 tasks, excluding the task of Reacher_ hard. The results involving SEGNN and Reacher_hard are deferred to Figure 4.
Training details. Each training run consumes 2M time steps (1000 steps per episode for 2000 episodes) for all the tasks except for Humanoid run and Humanoid stand, for which each training run consumes 5M steps. Evaluation is performed every 10000 steps by averaging the episodic rewards of 10 episodes, which is reported in our figures as the y-axis. all the figures, we plot the mean performance over 5 seeds together with the shaded regions which represent 95% confidence intervals. The training runs are computed by NVIDIA V100 single-GPU machines, each taking roughly 2 hours to finish IM training time steps for our method and all the baselines except for the SEGNN baseline, which takes roughly 70 hours finish 1M steps.
5.1 Comparison with standard RL and prior data augmentation methods
As SEGNN is prohibitively expensive in computation, we only run it on the task of Reacher_ the smallest one among all 10 tasks, and defer the results to Section 5.2. Figure presents the learning curves to compare our method against all other basclines on the rest 9 tasks. For our method, we present both Paug 0% and the task-specific best positive Paug E {25%, 50%, 75%, 100%} to separate the effects of our limb- babdd kinematic state representation and the data augmentation on top of it. We make the following observations from the results:
(i) DDPG is comparable to or significantly better than SAC in all 9 tasks except for Cheetah_run and Humanoid_ -unn which justifies our choice of DDPG the base RL algorithm.
(ii) As existing alternative data augmentation methods for state-based continuous control, the base- lines of GN and RAS do not offer significant improvement in data efficiency. They even introduce negative effects on performance on most tasks, except for Cheetah3D _rn and Humanoid_stad __and where RAS improves data efficiency over its based algorithm of DDPG.
(iii) With Pang = 0%, i.e., with the limb-based state representation alone, our method improves data efficiency over DDPG on the 6 tasks shown in the first rows in Figure 3, and in 5 out the tasks (excluding Hopper3D (hoperrD__hop...oo Paug 0% is comparable to the best Paug > 0%. On the rest of
8

---

4 tasks [Hopper3D_hor and 3 tasks in the last row), a positive Paug > 0 is crucial to attain the best performance among all the methods.
To summarize, our method reliably improves DDPG, the state-of-the-art RI. algorithms on the continuous control tasks. The improvements pecially on tasks with rich 3D DoFs and/or large numbers of state features: for example, the improvements on Hop- per3D run are more salient on its 2D variant Hopper run; our aggressive data augmentation (Poug 100%) neces- sary to achieve the best performance on Humanoid_ _un and
1000
remarkably significant, es-
800
SAC
Upisndee
400
DDPG + RAS
DDPG Dursip.. 0%)
200
DDPG + Ours 25%
SEGNN
25
50
75
100 125 150 175 200
Humanoid stand, the hardest two tasks where no baseline is able to learn on both.
Time step (x10f)
1000
YW
800
5.2 Comparison with equivariant agent architecture
800
Epionde
400
Employing equivariant neural networks such as SEGNN as the agent policy and critic architecture is an alternative method to exploit the Euclidean symmetry but at the same time more computationally expensive than data augmen- tation. As we are not able to afford to finish the SEGNN baseline for all tasks, we have run only for Reacher Reacher_hard, the smallest one by the number of limbs/joints. Figure 4 reports the learning curves of our method, SEGNN, and all other baselines on Reacher_iee __ard, with the x-axis being time steps and hours to compare data and computational efficiency, respectively. As the results show, SEGNN is the most data efficient, followed by data augmentation method (Paug 25%). both clearly outperforming the others. However, our method introduces minimal computation in addition its DDPG base algorithm, both taking roughly 2 hours to finish 1M steps after convergence, while SEGNN takes more than 70 hours to finish 1M steps and more than 10 hours to converge.
DDPG
200
SEGNN
60
Hours (for 1M time
Figure 4: Learning curves of data effi- ciency (top) and run time for IM steps in total (bottom) for our method and all baselines on Reacher_harr
5.3 Effect of Paug
In Sections 5.1 and 5.2, we have presented the learning curves of our method with Pang = 100%
and best task-specific Paug 0%). Here, we present further results detailing the effect of Paug. with the learning curves of all values Pang on all tasks given in Appendix B.1. We observe that: 1) On relatively simple tasks with 2D planer DoF's and small numbers of limbs/joints, Paug 0% does provide significant improvements over Pang 0% examples including Cheetah run as shown in Figure 5a; 2) Data augmentation (Pang 0%) is crucial best performance
hard tasks with 3D DoFs and large numbers of limbs/joints, examples including Hopper3D hhppepDDhpppphoo and Humanoid run in Figures 5b and Sc, respectively; and 3) The best Paug task-specific and not necessarily the largest possible Paug = 100%. For example, Pang = 100% hinders performance on Hopper3D_mh _hop but is the best on Humanoid run.
175
200
250
155
125
150
100
50%
100
75%
25
102%
10
100
203
110
150
703
50 102 155
205 750 100 10S 400 40D YO "Tmee step bx10")
Time step bx100))
Time step 1o3
(a) Cheetah run
(b) Hopper3D_ppppp _hop
(c) Humanoid run
Figure 5: Learning curves of our method on the effect of Pang on three sample tasks.
These observations underscore the task-specific nature of augmentation strategies, which aligns with findings from previous studies in the field of computer vision, where the idea data augmentation originated and has prevailed. Cubuk et al. [32] demonstrated that different datasets such as CIFAR- 10, SVHN, and ImageNet require distinct augmentation strategies. Similarly, Zoph et al. [33] showed significant differences in the optimal augmentation strategies for various object detection

---

250
Joint DeGod Mapp 0%
Jdine ressed,
150
175
Lime basod. dous 100%
700
Limb based, 1009s
[ 125
5500
8 1100
160 Pag 100%
100
Lim based. 50%
300
DEPG
100
10 TOS Tho 203 253 105 W0 to Ta
50 105 155
200 250
Time step 1.01
Time step 1x10)
"ime step (x100)
(a) Hopper3D_ hop
Humanoid run
Humanoid stand
Figure 6: Comparison with joint-based data augmentation three sample tasks.
tasks, highlighting the task-specific requirements of augmentation policies. Ho et al. [34] further emphasized this point in their study on Population Based Augmentation, demonstrating that the best augmentation policies vary across tasks, underscoring the need to tailor augmentation strategies to specific tasks and datasets.
5.4 Comparison with joint-based data augmentation
Indeed, one can perform S0₂(3) rotations on the original joint-based state representation. In the presented tasks, only the torso 1) has features for orientation (in its up DoFs) while features of other limbs (i > 1) are just angles and angular velocities of the hinges. Therefore, under a rotation, only the torso's 's orientation features are changed, while other features stay unchanged. Therefore, we hypothesize this data augmentation by rotating original joint-based features would bring little benefit the tasks. This is because torso features make up only a fraction all features when the number of limbs is large (which is the case in these tasks). Our munb-ased representation instead rotates all limbs provide richer augmentation. The hypothesis is supported by results of prior work (e.g.. see Figure 16 in Corrado and Hanna [18]). We here also conduct comparison, with results in Figure 6 showing that joint-bacc based data- ailla-augmeatttion less efficient than our limb-based method.
6 Conclusion
We have motivated, formalized, and developed the idea and method of a new data augmentation method that aims to improve the performance of RL, as measured by its data efficiency and asymptotic reward, for state-based continuous control. The key idea of our data augmentation method is to create additional training data by performing Euclidean transformations on the original states, which justified by the Euclidean symmetries inherent in the continuous control tasks. To make the states more amenable to Euclidean transformations, we turn to state representation based on limbs' kinematic features, rather than on joints' configurations as done by default prior works. Our new method significantly improves the performance for wide range of state-based continuous control tasks, especially for tasks with rich 3D motions and large numbers of limbs and joints.
Limitations. Our work focuses on robot locomotion tasks as instances of continuous control, which clearly exhibit Fuclidean symmetries. Other robotics tasks (e.g., navigation) and many applications that operate in the 3D physical space should also exhibit Fuclidean symmetries. There are indeed continuous control tasks that might not exhibit Euclidean symmetries, such as those in electrical and power engineering, which we do not consider. We observe two limitations that would inspire future work: 1) To obtain the best performance, our data augmentation method needs task task-specific turning of its hyperparameter, Paug. that controls the proportion of data to be transformed for learning. Future work in this direction needed to ease task-specific turning, either by adaptive tuning strategies [35] or training techniques to make learning more robust and less sensitive to the hyperparameter [36]; 2) The proposed method requires knowledge and annotation strict Euclidean symmetries in the continuous tasks, which might not be easily available especially when the external forces are noisy and hard to detect, e.g., including wind conditions the wild field along with gravity. This prompts future work of automatic discovery and exploitation of irregular, approximate Euclidean symmetries.
10

---

Acknowledgments and Disclosure of Funding
The authors acknowledge funding support from Qi Zhang's NSF CAREER award 2237963. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do necessarily reflect the views of the sponsors. The authors thank the anonymous reviewers for their insightful and constructive reviews.
References
[1] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised repre- sentations for reinforcement learning. In International conference machine learning, pages 5639-5650. PMLR, 2020.
[2] Misha Laskin. Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884-1999 19895, 2020.
[3] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021.
[4] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. International Conference on Learning Representations, 2022.
[5] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.00807 2018.
[6] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[7] Balaraman Ravindran and Andrew G Barto. Symmetries and model minimization in markov decision processes, 2001.
[8] Elise Van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems, 33:4199-4210, 2020.
[9] Dian Wang, Robin Walters, and Robert Platt. International Conference Learning Representations, 2022.
SQuui-equivariant reinforcement learning. In
[10] Sahand Rezaci-Shoshtari, Rosie Zhao, Prakash Panangaden, David Meger, and Doina Precup. Continuous mdp homomorphisms and homomorphic policy gradient. Advances in Neural Information Processing Systems, 35:20189 09229994, 2022.
[11) Greg Broekman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint /606.006000000 54040 2016.
[12] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. et al. Deepmind control suite. arXiv preprint 11X1.0:80110000 2018
[13] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509. 02971, 2015.
[14] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in neural information processing systems, 34:3680-3699 3693, 2021.
[15] Sicong Liu, Xi Sheryl Zhang, Yushuo Li, Yifan Zhang, and Jian Cheng. On the data-efficiency with contrastive image transformation in reinforcement learning. In The Eleventh International Conference Learning Representations, 2023.

---

[16] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. In A. Beygelzimer, Y. Dauphin. P. Liang, J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[17] Nicholas E Corrado, Yuxiao Qu, John U Balis, Adam Labiosa, and Josiah Hanna. Guided data augmentation for offline reinforcement learning and imitation learning. arXiv preprint arXiv:2310.18227, 2023.
[18] Nicholas Corrado and Josiah P. Hanna. Understanding when dynamics-invarte data aug- mentations benefit model-free reinforcement learning updates. The Twelfth International Conference on Learning Representations, 2024.
[19] Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems, 33:3976-3990, 2020.
[20] Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model- based coun- terfactual data augmentation. Advances in Neural Information Processing Systems, 35:18143 18156, 2022.
[21] Elise van der Pol, Herke van Hoof, Frans A Olichoek, and Max Welling, Multi-agent MDP homomorphic networks. In International Conference on Learning Representations, 2022.
[22] Farzad Abdolhosseini, Hung Yu Ling, Zhaoming Xie, Xue Bin Peng, and Michiel Van de Panne. On learning symmetrie locomotion. In Proceedings of the 12th ACM SIGGRAPH Conference on Motion, Interaction and Games, pages 1-10, 2019.
[23] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference machine learning, pages 387-395. Pmir, 2014.
[24] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International conference on machine learning. pages 1282-1289. PMLR, 2019.
[25] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: simple technique for generalization in deep reinforcement learning. In International Conference on Learning Representattonss 2020.
[26] Runfa Chen, Jiaqi Han, Fuchun Sun, and Wenbing Huang. Subequivariant graph reinforcement learning in 3d environments. In International Conference on Machine Learning, pages 4545- 4565. PMLR, 2023.
[27] Emanuel Todorov, Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033 IEEE, 2012.
[28] Simulation MuloCo Documentation. programming! simmlationnnnmmmmmmmmmmmmmmmmiit html.
[29] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error
in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018
[30] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[31] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve c(3) equivariant message passing. In International Conference Learning Representations, 2022.
[32] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 113-123, 2019.
12

---

[33] Barret Zoph, Ekin D Cubuk, Golnaz Ghiasi, Tsung- Yi Lin, Jonathon Shlens, and Quoe V Le. Learning data augmentation strategies for object detection. Computer EsiionEECY' 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pages 566-583 Springer, 2020.
[34] Daniel Ho, Eric Liang. Xi Chen, lon Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International conference machine learning, pages 2731-2741. PMLR, 2019.
[35] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprini arXiv: 18/2.0595, 2018.
[36] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 .44144, 2023.
13

---

A Implementation details
A.1 Task details
Kinematic features. Table lists for all the 10 tasks the kinematic parameters including number of limbs n, number DoFs of the torso limb di, numbers of 2-, and 3-DoF joints DI.DoF No2.DoF and ng-Doo. There three possible values for d1:
dj 0. The torso is fixed, e.g., in Reacher_hare hard.
di 3. The torso has all 3 Dolss: moving along the r-axis, moving along the z-axis, and rotating planar tasks.
about the y-axis, which is the case
The torso has all 6 DoFs, which is the case in 3D tasks.
These parameters are sufficient to recover other kinematic parameters of task. For example, action size m E:>1 d: VI--De + 1 3 according to (2), the size of our limb-based kinematic features of a task should be 3 3 + 3 + 3n + 3(2m + 3n, )
A caveat is that, for the number of limbs in Table 1, do not count the bodies that do not have any DoF with respect to its parent, examples including the finger with respect to wrist in Reacher_haid An exception the torso in Reacher hard, where we do count it in Such bodies welded its parent. By doing so, we keep the relationship of n = TI-DDFF + 112-DoF + D...De + in Table 1. In our implementations, however, we do include the kinematic features of these bodies.
Table 1: Kinematic parameters of the tasks. n: number of limbs. d1: number of DoFs of the torso limb (i 1). D.F. DoP. Ro..DeF number of 1-, 2-, 3-DoF joints.
Task
dj
Ri-Doo
R2-DoF
713-DoF
Cheetah run Cheeetah3D run
6
7
6
0)
0
6
Hopper_hop Hopper3D_hap __hop
4
0
0
4
Walker_run run
6
0
Walker3D run
6
0
0
6
Quadruped run
13
8
4
0
Reacher_har hard
0
2
0
0
Humanoid run/stand
13
5
5
Sensory observations.
Table 2 lists the sensory observations for all the 10 tasks.
Table 2: Sensory observations in the tasks.
Sensory Observations
Task
Number of Invariant Features
Number of Equivariant Features
Cheetah run
Cheetah3D run
Hopper hop
2:two foot touch sensors
Hopper3D hop
2:two foot touch sensors
Walker run
1:the height of the torso
Walker3D run
1:the height the torso
Quadruped_run
4: the 4 foot torque
4: the 4 foot force
:: the dot-product of the torso z-axis and the global 7-axis 1ee gyroscope
1: the acceleration
4:the 4 egocentric state
Reacher hard Humanoid run/stand
:: the vector from target to finger
the z-projection of the torso orien- tation matrix
I: the velocity of the center- of-mass
4: the end effector posi- tions in egocentric frame
14

---

A.2 DDPG hyperparameters
Table 3 presents the full list of DDPG hyperparameters used in our method and baselines with DDPG as the base RL algorithm, including standard DDPG, DDPG GN, DDPG RAS, and DDPG Ours. The hyperparameters and other implementation details of the SEGNN- baseline deferred to Section A.3.
Table 3: DDPG hyperparameters used in our experiments.
Hyperparameter
Setting
Learning rate Optimizer
le-4
Adam
n-step return Mini-batch size
3
256
Actor update frequency
2
Target networks update frequency
2
Target networks soft-update
0.01
Target policy smoothing stddev. clip
0.3
MLP hidden size
256
Replay buffer capacity
10"
Discount y Seed frames
0.99
4000
Exploration steps Exploration stddev. schedule Action repeat
2000
linear 0.1, 1e6)
A.3 SEGNN
SEGNN is recently developed neural network architecture designed to preserve the invariance and equivariance of features under 3D transformations such as rotations, translations, and reflections. operates an Euclidean graph which is point cloud, where each point (node) comprises positions, node features and edge features, ensuring that the output Euclidean graph also maintains these properties. Below, we illustrate its application in the Euclidean graphs in the task Reacher hard.
The task has rotation-equiivarinee in the ry plane, and translation-invariange due to the hinge fixed at the origin. The state contains The state contains the positions of the target, finger, hand, arm, and root (fixed origin), and the velocities of the finger, hand, arm, and root (fixed constant 0). All these features are equivariant and they are transformed into the state-action based and state based Euclidean graphs for critic and actor, respectively. Specifi- cally, the point set in the point cloud is V = (target, finger). The node feature for pointfnger: (a, ||a|]]. The node feature poode puode,target target for pointtarget is {0). The node attribute contains the node type: Vi E V, frode = [node_ (oo__ttyp_))____ The Observation based Euclidean graph is the same as the state-action based Euclidean graph, except that action not included in the node feature.
Reacher hard:
finder ([V,, ; II (e hand arm. U {x,,, [[xx]|l])//(aann.emmm.oo
feature
15

---

Table 4: Hyperparameters for our SEGNN-based DDPG implementation for Reacher_ hard.
Hyperparameter
Setting
Learning rate Optimizer
5e-5
Adam
n-step return Mini-batch size
3
256
Actor update frequency
2
Target networks update frequency
Target networks soft- soft-update
0.01
Target policy smoothing stddev. clip
0.3
SEGNN hidden size
64
Replay buffer capacity Discount ? Seed frames
106
0.99
4000
Exploration steps
2000
Exploration stddev. schedule Action repeat
linear 0.1, le6)
16

---

B Supplemental results
B.1 Complete results on the effect of Pang
Figure 7 presents the learning curves of all of our choices of Pang for all the 10 tasks, completing the results in Figure 5.
300
500
250
220
25%
300
110
50%
200
1.00
100
50
(x300%)
step bx30f)
(x10%)
(a) Cheetah_rue run
(b) Cheetah3D run
(c) Hopper_hop
800
700
1.0
Esuzde venee
E60
1000
150
400
00
100
300
100
20
Time step 201)
Time step 10
Time step (x10t)
(d) Hopper3D hop
(e) Walker run
(f) Walker3D run
1000
175
1.50
800
6:0
100
400
.00
200
200
25
103
155 200 750 TO 40 TO
Time step Ox105
Tiim step bx3d))
"imee step (x10))
(g) Reacher hard
(h) Quadruped run
(i) Humanoid run
5.00
50 100 155 100 150 300 350 400 480 500 Time slep (x))
Hlumanoid_starl stand
Figure 7: Learning curves of our method on the effect of Paug on all 10 tasks.
B.2 SEGNN with single-point Euclidean graph
Figure 8 presents the results of SEGNN on Reacher hard and Cheetah3D run with Euclidean graph of a single point which contains all the state features. It shows that SEGNN performs much worse than the MLP--ased architectures without a good design of the graph (point set).
6:00
1303
5:00
Este0de eXAMeE
4:00
002
DDPS
DDPS DDPS GustAw 25%1 aDec SCOW
100
Dbes a Dusti 2531
DDPS SEERN
25 34
1:0 12% 155 17t 700 Time step (x10fs
150 115 Time step (4109)
150 175 200
(a) Cheetah3D run
(b) Reacher hard
Figure 8: Learning curves single-point SEGNN, standard DDPG, and our method.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make it clear in our abstract and introduction that this paper provides a novel data augmentation method for reinforcement learning-based continuous control. This accurately reflects the paper's contributions and scope as described in Sections and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
include a paragraph labeled "Limitations" in Section discuss the Answer: [Yes]
Justification:
limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
18

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper include enough details for reproducibility in Section 5 and Appendix
A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide our code in the supplemental material along with instructions to reproduce the main experimental results, which will be open-sourced upon the acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Baaili//guides/CoeSSummii oollliiiiiiioooooriiioico for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: gwwd////e/////gg CCoeae/mdddSSbaaaaa
/ //npsscc...
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper include enough details necessary to understand the results in Section and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
As described in Section 5, the paper reports error bars corresponding 95%
CI.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section describes the computer resources needed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / wttppppsccccpppcccpecccpcccp htte:/pullc//tihcceecccoo iembbiii/Etccaaaddeeeineee
ceGuideliness
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and are certain that the paper conforms with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed, the paper addresses the lcarning problem in simulated continuous control tasks, which does not raise immediate societal impacts.
Guidelines:
21

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: That the paper poses such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper properly credits the original creators as cited in Section 5.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide well-documented code,
asset, in the supplemental
material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: he paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "T8YaeUIAQE", "reviewer": "Reviewer_aR8o", "summary": "This work proposes a novel data augmentation approach that leverages Euclidean symmetry for continuous control to improve the efficiency and performance of RL algorithms. The authors integrated their approach with DDPG and performed a series of comparison against the vanilla SAC algorithm, other augmentation techniques, and other equivariant methods. The results show some improvements in performance across a range of continuous control tasks, especially those with rich 3D motions and large number of limbs and joints.", "questions": "1. In Section 5.1, the authors say \"To summarize, our method reliably improves DDPG and SAC, the state-of-the-art RL algorithms on the continuous control tasks.\" However, it looks like only vanilla SAC is used in this work for comparison. Hence, the mention of improving SAC should be removed.\n\n2. In line 194 p^y_1 should be  p^z_1.", "limitations": "Technical limitations are highlighted in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach improved the learning efficiency of RL algorithms (demonstrated with DDPG in this work) without the need of changes to the algorithm, which lowers the barrier for adaptability for the community.\n2. Euclidean transformations maintain the inherent physics of the task, ensuring that the augmented data are still representative of realistic scenarios. This preservation is crucial for the relevance and usefulness of the augmented data.", "weaknesses": "1. This method is primarily applicable to tasks with clear Euclidean symmetries. So, it seems that in environments where such symmetries are not evident or relevant, the approach may not be applicable or effective. While the 'Limitation' section briefly touches upon this issue, it is not clear what categories of continuous control problems this approach will be applicable to.  \n2. The effectiveness of Euclidean transformations may vary depending on whether the task is set in a 2D or 3D space, with potentially limited benefits in more constrained settings (like planar movements).\n3. As the 'Limitation' section suggests, practical application of the framework will be difficult due to the task-specific hyperparameter tuning requirements. While the author suggests adaptive hyperparameter tuning as a potential future work, some insights into what characteristics of the environment dynamics dictate the optimal choice of hyperparameters such as \\rho_{aug} would be helpful. Also, in the context of practical application, switching from joint-based to limb-based configurations complicate the state representation and requires significant changes to the training environment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review! We are glad to provide a response and look forward to follow-up discussions.\n\n**On Weakness 1**  \nOur work focuses on robot locomotion tasks as instances of continuous control, which clearly exhibit Euclidean symmetries. Other robotics tasks (e.g., navigation) and many applications that operate in the 3D physical space should also exhibit Euclidean symmetries.\nWe agree that there are continuous control tasks that might not exhibit Euclidean symmetries, such as those in electrical and power engineering.\nWe will update our \"Limitation\" section and other parts of the paper to clarify this. \n\n**On Weakness 2**  \nWe agree that the effectiveness of Euclidean transformations depends on the task, and our results indicate that it is even more subtle than just 2D vs 3D, for which we provide a more detailed interpretation below. In short, Euclidean transformations are effective if the task itself exhibits corresponding symmetries; otherwise, one should not expect these transformations to improve the performance in the first place.\n\nOur experiments include 10 tasks, 9 in Figure 3 and the 10th in Figure 4. \n   \n4 out of the 10 tasks are 2D: Figure 3’s (a)(c)(e) and Figure 4. For (a)(c)(e), $\\rho_{\\rm aug}$ > 0% is no better than rho_aug = 0%; for Figure 4 (Reacher_hard), $\\rho_{\\rm aug}$ > 0% is better. This can be explained by the nature of the tasks: \n\n- In (a)(c)(e), a robot learns to walk/run forward with its “legs” in the 2D xz-plane, and there are no rotation symmetries within the plane itself. Our method adds the y-axis and performs rotations about the z-axis, which is valid but does not generate more data within the xz-plane that would be useful for the original 2D task. Therefore, it is expected that $\\rho_{\\rm aug}$ > 0% will provide limited benefit for these tasks.\n\n- In Figure 4 (Reacher_hard), a 2-limb robot is placed on the xy-plane (ground) and is tasked to rotate about the z-axis to reach a target point on the xy-plane with its tip. There are rich rotation symmetries within the 2D xy-plane, and our data augmentation generates more data therein. Therefore, it is expected that $\\rho_{\\rm aug}$ > 0% will be beneficial.\n\nFor the other 6 tasks which are 3D, $\\rho_{\\rm aug}$ > 0% is slightly better than $\\rho_{\\rm aug}$ = 0% (especially in early training) in Figure 3’s (b)(f) and is much better in the other 4 tasks. This is because these tasks all have rotation symmetries about z-axis, so our data augmentation is beneficial. Note that these tasks include the 3D version of Figure 3’s (a)(c)(e).\n\n**On Weakness 3 - tuning $\\rho_{\\rm aug}$**   \nPer our response to Weakness 2, our results have provided the following insights:\nTasks with richer symmetries often benefit from larger values of $\\rho_{\\rm aug}$ (e.g., $\\rho_{\\rm aug}$=100% is best for Humanoid tasks).\n\n**On Weakness 3 - limb-based vs joint-based configurations**   \nAlthough simulators like MuJoCo often use joint-based configurations as default, both configurations are convenient to obtain. Given a joint-based configuration, the computation of its corresponding limb-based configuration is known as “forward kinematics”, which is well-studied in robotics and implemented in simulators such as MuJoCo. We did use MuJoCo’s original APIs for forward kinematics. Therefore, we argue using limb-based configurations is not a significant issue in practice.\n\n**On Question 1**  \nThank you for the suggestion. We will remove \"SAC\" in the statement. \n\n**On Question 2**  \nThank you. We will fix it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nvjm2mdg14", "reviewer": "Reviewer_X7ZX", "summary": "This paper proposes a data augmentation technique that leverages Euclidean symmetries (e.g. rotational symmetry) in a task's dynamics to generate augmented data. When a task's state features do not have such symmetries, the paper also discusses how to define a new state representation with these symmetries (so that data augmentation can be applied). Empirically, the paper shows that RL is more data efficient with this data augmentation technique than without it on various DMControl tasks.", "questions": "1. Line 40-43: Could the authors elaborate on what “uncorrelated” means here?\n\n4. Line 159: Does anything change if the morphology tree contains a cycle?\n    \n5. Line 272: What does it mean to keep $a_t$ and $r_t$ invariant?", "limitations": "The paper addresses limitations, though I think an additional limitation should be emphasized: not only may itbee difficult to specify a symmetry, symmetries might not even exist in some tasks. \n\nI don't believe this limitation takes anything away from the paper though; it's a core limitation of most non-perturbation-based data augmentation methods, and it's fine if these methods only apply to some tasks. That's simply the nature of data augmentation. Thankfully, in many real-world tasks (especially robotics tasks where you have an agent acting in 3D space) a human *can* identify symmetries. Pitis et. al [1] and Corrado et. al [2] discuss this point too.\n\n[1] Pitis et. al. “Counterfactual Data Augmentation using Locally Factored Dynamics.” NeurIPS 2020.\n\n[2] Corrado & Hanna. \"Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates.\" ICLR 2024.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper focuses on a class of augmentations that have been under-studied (compared to visual augmentations)\n2. Redefining a task's state representation to enable data augmentation is an interesting concept to me.\n3. Comparing the proposed augmentation strategy with an RL agent that learns with an equivariant network architecture was useful; I've always wondered how these two different approaches compared.", "weaknesses": "1. **The story seems improperly situated in the data augmentation literature.** While it is true that most prior data augmentation works focus on perturbation-based augmentations, many prior works have exploited task symmetries and invariances to generate additional data that agrees with the task's dynamics and reward function [1-6]. Corrado et. al [1] calls these dynamics-invariant augmentations.\n\n2. **It’s unclear if this change of representation is necessary for data augmentation.** Corrado et. al [1-2] perform data augmentation using SO(3) rotations in MuJoCo tasks similar to those studied in this paper *without* changing the task’s state features.\n\n3. **Weak empirical results.** In 5/9 tasks, DDPG + data augmentation performs just as well as DDPG without data augmentation. With 5 seeds per curve, I have low confidence in the observed benefits in the remaining tasks; RL is notoriously high variance, and the variance between runs is enough to create statistically different distributions. 95% confidence belts assume the distributions of returns at each evaluation point are normally distributed, which is likely not the case. \n\n[1] Corrado & Hanna. \"Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates.\" ICLR 2024.\n\n[2] Corrado et. al. \"Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning.\" arXiv:2310.18247\n\n[3] Pitis et. al. “Counterfactual Data Augmentation using Locally Factored Dynamics.” NeurIPS 2020.\n\n[4] Pavlov et. al. \"Run, Skeleton, Run: Skeletal Model in a Physics-Based Simulation.\" AAAI 2018.\n\n[5] Abdolhosseini et. al. “On Learning Symmetric Locomotion.” ACM SIGGRAPH 2019.\n\n[6] Adrychowicz et. al. \"Hindsight Experience Replay.\" NeurIPS 2017.\n\n[7] Henderson et. al. \"Deep Reinforcement Learning that Matters.\" AAAI 2018\n\n**Minor Comments**\n1. I think section 3.2 can be omitted; the augmentation framework described in section 4.3 can be applied to any off-policy RL algorithm.\n    \n6. Line 271: typo, I think the second $s_t$ should be $s_t\\prime$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review! We are happy to provide a response. We hope it can trigger your re-evaluation and look forward to follow-up discussions.\n\n**On Weakness 1**   \nThank you for suggesting the papers as they are indeed related to our work.\nAs some of the papers are also mentioned by Reviewer 5Bxo, please refer to our global response that compares our work with those papers. The comparison clarifies our different and orthogonal contributions.\n\n **On Weakness 2**   \nIndeed, one can perform SO(3) rotations on the original joint-based state representation. In our tasks, only the torso ($i=1$) has features for orientation (in its up to 6 DoFs) while features of other limbs ($i>1$) are just angles and angular velocities of the hinges. Therefore, under a rotation, only the torso’s orientation features are changed, while other features stay unchanged. This is what prior work such as Corrado et. al. did.\n\nHowever, we hypothesized this data augmentation by rotating original joint-based features would bring little benefit to locomotion tasks. This is because torso features make up only a fraction of all features when the number of limbs is large (which is the case in our tasks). Our limb-based representation instead rotates all limbs to provide richer augmentation.\n\nOur hypothesis is supported by results of Corrado & Hanna [1] (see Figure 16 of [1]: confidence intervals are overlapped for Ant; rotation underperforms than no-augmentation for Humanoid).\nThis is also evidenced by our added results in Figure 3 in the rebuttal pdf: we have added curves corresponding to performing rotation data augmentation on original state features, which is less performant than ours.\n\n **On Weakness 3 - effectiveness**  \nWe first clarify that we have performed 10 tasks, 9 in Figure 3 and the 10th in Figure 4.\nWe here provide a more detailed interpretation of the results to explain the effectiveness of our method.\n\n4 out of the 10 tasks are 2D: Figure 3’s (a)(c)(e) and Figure 4. For (a)(c)(e), $\\rho_{\\rm aug}$ > 0% is no better than $\\rho_{\\rm aug}$ = 0%; for Figure 4 (Reacher_hard), $\\rho_{\\rm aug}$ > 0% is better. This can be explained by the nature of the tasks: \n\n- In (a)(c)(e), a robot learns to walk/run forward with its “legs” in the 2D xz-plane, and there are no rotation symmetries within the plane itself. Our method adds the y-axis and performs rotations about the z-axis, which is valid but does not generate more data within the xz-plane that would be useful for the original 2D task. Therefore, it is expected that $\\rho_{\\rm aug}$ > 0% will provide limited benefit for these tasks.\n\n- In Figure 4 (Reacher_hard), a 2-limb robot is placed on the xy-plane (ground) and is tasked to rotate about the z-axis to reach a target point on the xy-plane with its tip. There are rich rotation symmetries within the 2D xy-plane, and our data augmentation generates more data therein. Therefore, it is expected that $\\rho_{\\rm aug}$ > 0% will be beneficial.\n\nFor the other 6 tasks which are 3D, $\\rho_{\\rm aug}$ > 0% is slightly better than $\\rho_{\\rm aug}$ = 0% (especially in early training) in Figure 3’s (b)(f) and is much better in the other 4 tasks. This is because these tasks all have rotation symmetries about z-axis, so our data augmentation is beneficial. Note that these tasks include the 3D version of Figure 3’s (a)(c)(e).\n\nTo summarize, we did not cherry pick tasks where our method outperforms others; instead, we sampled a range of tasks with various degrees of symmetries to show when and why our method can bring benefits.\n\n **On Weakness 3 - seeds and confidence intervals**   \nFor the 4 tasks where $\\rho_{\\rm aug}$ > 0% is better, we have completed another 5 seeds during rebuttal.\nWith the 10 seeds,  Figure 2 in the rebuttal pdf updates the training curves with mean+95% CI. \nFigure 2 also gives inter-quartile means (IQMs) with 95% bootstrap confidence intervals, which is more robust to outliers  (used in prior work like Corrado et. al. [2]).\nWe believe these results give enough confidence on the effectiveness of our method. \n\n **On Minor Comment 1**   \nThank you for the suggestion. We will consider removing Section 3.2 in our revision.\n\n **On Minor Comment 2**  \nThank you. We will fix it.\n\n **On Question 1**   \nWe mean, although the original transition comes from the ground truth dynamics and reward functions, the perturbed transition does not necessarily. In this sense, they are “uncorrelated”. In the words of Corrado et. al., perturbation is not a Dynamics-Invariant data augmentation.   \n\n **On Question 2**    \n For morphology trees containing cycles, our key idea and methodology, i.e., ${\\rm SO}_{\\vec{g}}(3)$ data augmentation on limb-based representation, still applies well. One just needs to identify the kinematic features therein and perform appropriate transformations.\nWe focus on no-cycle trees because most locomotion tasks (including all of our tasks) do not contain a cycle, and the torso/base often serves as the tree root. In our tasks, the root is special in the sense that it can have up to 6 DoFs, while other nodes only have hinge-like DoFs. Again, we focus on this case to make our method description clear, but our methodology applies to cycles.  \n\n **On Question 3**  \nThat simply means we do not change them.\nSince actions a_t are scalar torques, they should not change under rotations.\nRewards should also stay the same under rotations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H3NVh7p6ZI", "reviewer": "Reviewer_rWGW", "summary": "This paper introduces a novel data augmentation strategy tailored for reinforcement learning (RL) agents operating in state-based continuous control environments. The method leverages limb-based state features rather than joint-based configurations, allowing for more effective augmentation. Experiments conducted on various tasks from the DeepMind Control Suite demonstrate significant improvements in data efficiency and performance over standard RL algorithms and existing augmentation methods.", "questions": "Could the authors provide more details on how to tune the various parameters involved in the proposed method, such as the choice of Euclidean transformations and the proportion of augmented data?", "limitations": "The authors acknowledge the need for task-specific tuning of the hyperparameter and the requirement for knowledge of strict Euclidean symmetries. They have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a unique approach to data augmentation for RL in state-based continuous control, moving away from traditional perturbation methods to Euclidean transformations. This approach is innovative and addresses the limitations of existing methods.", "weaknesses": "Firstly, I am not an expert in this research field. I think this paper lacks some deeper theoretical support. The scenario covered by Theorem 1 is overly simplistic. The proposed method involves additional computations for applying Euclidean transformations and managing the limb-based state features. The paper could benefit from a more detailed discussion on the computational overhead and potential optimizations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review! We are glad to provide a response and look forward to follow-up discussions.\n\n**On Weaknesses - theoretical support**   \nThe paper does not state a Theorem 1. Can you further clarify your concern? Thanks. \n\n**On Weaknesses - computational overhead**   \nOur method requires minimal additional computation. The only additional computation is transforming a subset of transitions in the mini-batch, as described in Section 4.3. Note that, compared to the standard method with no data augmentation, we keep the same batch size and gradient update to ensure no additional computation is required. The transformation is essentially matrix multiplication (involving 3D rotation matrices), which incurs negligible overhead in the training process, as we have shown in Figure 4 (bottom).\n\n**On Questions - choices of Euclidean transformations**   \nThe available Euclidean transformations are not tunable, because one should only choose transformations that respect the symmetries in the task of interest.\nWe focus on rotation symmetries about the z-axis (${\\rm SO}_{\\vec{g}}(3)$) because it is the only available symmetries in our tasks.\nThe other Euclidean transformations, i.e., translations or reflections, are not available because:\n\n- The robots in the tasks use egocentric representation (line 256) and therefore it’s already translation-invariant.\n\n- Reflections require gait symmetry (e.g., the left leg has the same length, stiffness, etc.), which the robots in the tasks do not necessarily have. Some prior work modified the robot to enforce gait symmetry before applying reflections (e.g., see Table 1 of Corrado & Hanna labeled as [1] by Reviewer X7ZX), but we did not do it.   \n\n**On Questions - tuning hyperparameters**   \n$\\rho_{\\rm aug}$ is our only additional hyperparameter on top of an off-the-shelf algorithm like DDPG. Its tuning is standard, no different than other hyperparameters:\n\n- We did grid search, which is standard and the same way as tuning other DDPG hyperparameters\n\n- We have discussed the effect of $\\rho_{\\rm aug}$ in Section 5.3, which gives insights on tuning strategy. In short, tasks with richer symmetries benefit from a larger $\\rho_{\\rm aug}$.\n\n- An automatic tuning method might be beneficial, which we leave for future work (Section 6).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HZafxp6Mlb", "reviewer": "Reviewer_5Bxo", "summary": "This paper introduces a novel data augmentation method for reinforcement learning in continuous control tasks. The key innovation is leveraging Euclidean symmetries inherent in these tasks by applying rotational transformations to the original states. The authors propose using a limb-based state representation instead of the standard joint-based one to make states more amenable to these transformations. Integrated with the DDPG algorithm, this method is evaluated on 10 tasks from the DeepMind Control Suite, demonstrating improvements in data efficiency and performance, especially for complex 3D tasks. The approach outperforms existing data augmentation techniques and proves computationally efficient compared to alternative methods.", "questions": "1. There appears to be a typo in line 271 - it should likely read \"transform states $s_t$ and $s_{t+1}$\".\n2. The description of rotations in Section 4.3 is confusing. Why are there two separate rotation steps mentioned?\n3. Some recent state-based data augmentation methods are not well cited in the related work, including:\n     - Hindsight experience replay (Andrychowicz et al., 2017)\n     - Counterfactual data augmentation (Pitis et al., 2020)\n     - MoCoDA (Pitis et al., 2022)\n     - Guided Data Augmentation (GuDA) (Corrado et al., 2024)\nParticularly, GuDA, while focusing on offline-RL and including human guidance, describes a method for state-based data augmentation, including rotation, transformation, and reflection. Discussing these works could strengthen the paper's positioning in the field.\n4. The paper focuses solely on the DDPG algorithm. Exploring compatibility with other popular RL algorithms would increase impact.", "limitations": "The authors adequately address limitations, noting the need for task-specific tuning and knowledge of strict Euclidean symmetries. They provide constructive suggestions for future work. Regarding societal impact, while the immediate concerns are limited for simulated tasks, a brief discussion on long-term implications for robotics, especially on real robots, could be valuable.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality:\n- The idea of leveraging Euclidean symmetries for data augmentation in state-based RL is novel and underexplored.\n- The limb-based state representation is an innovative solution enabling effective Euclidean augmentation.\n\nQuality:\n- Comprehensive experiments on 10 continuous control tasks, including 2D and 3D variants.\n- Thorough ablation studies on the effect of augmentation ratio.\n- Comparison against relevant baselines including standard RL algorithms and previous augmentation methods.\n\nClarity:\n- The paper is well-structured.\n- Limitations are discussed openly.\n\nSignificance:\n- The method shows substantial improvements on challenging 3D tasks like Humanoid_run where standard methods struggle.", "weaknesses": "1. The paper doesn't clearly address how the method handles environments where Euclidean data augmentation might violate constraints (e.g., obstacles in AntMaze from MuJoCo).\n2. The data composition process is not fully explained. Section 4.3 doesn't specify if $B_\\text{aug}$ transitions are added to $B$ or substitute points in $B$. \n3. The study is limited to 0-100% augmentation, while exploring larger multipliers could be insightful as one transition can be rotated with different angles\n4. The focus is primarily on rotation transformations. Exploring other transformations like translation and reflection could provide a more comprehensive analysis.\n5. In 5 out of 9 tasks, $ρ_\\text{aug}$ = 0% achieves the best performance, suggesting the limb-based representation may be more critical than the data augmentation itself. However, the choice of limb-based representation seems motivated primarily by enabling Euclidean DA, which may limit generalizability. In addition, the advantages of limb-based over joint-based representations are not thoroughly justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review! We are glad to provide a response to address your concerns and look forward to follow-up discussions.\n\n**On Weaknesses 1**   \nOur method (Euclidean rotations) can be straightforwardly applied to constraints like obstacles: one just needs to include constraints information in the task features (see our line 186, Section 4.1). \nFor example, in AntMaze, the robot is placed in a maze to navigate from a start point to an end point. Because the shape of the maze (i.e., the wall obstacles) is fixed, we can represent its shape using a directional vector and transform it (e.g., rotate) during data augmentation.\nSee Figure 1(c) in our rebuttal pdf for an illustration.\n\n**On Weaknesses 2**   \nWe here clarify that $B_{\\rm aug}$ transitions are transformed to *substitute* the original data points. This ensures that, compared to no data augmentation, we keep the same batch size and gradient update to ensure fair comparison.\nWe will make this clear in our revision.\n\n**On Weaknesses 3**   \nAs we have clarified for Weakness 2, the upper bound of $\\rho_{\\rm aug}$ is 100% because we do substitution.\nUsing multipliers larger than 100% needs to increase batch size, which we would avoid to ensure fair comparison with the no-augmentation baseline.\nNote that, even with substitution, a transition can still be rotated with different angles in different gradient update iterations.  \n\n**On Weaknesses 4**   \nOne should only choose transformations that respect the symmetries in the task of interest.\nWe do only rotations about the z-axis (${\\rm SO}_{\\vec{g}}(3)$) because it is the only valid transformation respecting the symmetries in our tasks.\nThe other Euclidean transformations, i.e., translations or reflections, are not valid because:\n    \n- The robots in the tasks use egocentric representation (line 256) and therefore it’s already translation-invariant.\n   \n- Reflections require gait symmetry (e.g., the left leg has the same length, stiffness, etc.), which the robots in these MuJoCo tasks do not necessarily have. Some work modified the robot to enforce gait symmetry before applying reflections (e.g., see Table 1 of Corrado & Hanna labeled [1] by Reviewer X7ZX), but we did not do it.\n\n**On Weaknesses 5 - performance**   \nWe first clarify that we performed 10 tasks, 9 in Figure 3 and the 10th in Figure 4.\nWe provide a more detailed interpretation of the results to explain the effectiveness of our method.\n\n4 out of the 10 tasks are 2D: Figure 3’s (a)(c)(e) and Figure 4. For (a)(c)(e), $\\rho_{\\rm aug}$ > 0% is no better than $\\rho_{\\rm aug}$ = 0%; for Figure 4 (Reacher_hard), $\\rho_{\\rm aug}$ > 0% is better. This can be explained by the nature of the tasks: \n\n- In (a)(c)(e), a robot learns to walk/run forward with its “legs” in the 2D xz-plane, and there are no rotation symmetries within the plane itself. Our method adds the y-axis and performs rotations about the z-axis, which is valid but does not generate more data within the xz-plane that would be useful for the original 2D task. Therefore, it is expected that $\\rho_{\\rm aug}$ > 0% will provide limited benefit for these tasks.\n\n- In Figure 4 (Reacher_hard), a 2-limb robot is placed on the xy-plane (ground) and is tasked to rotate about the z-axis to reach a target point on the xy-plane with its tip. There are rich rotation symmetries within the 2D xy-plane, and our data augmentation generates more data therein. Therefore, it is expected that $\\rho_{\\rm aug}$ > 0% will be beneficial.\n\nFor the other 6 tasks which are 3D, $\\rho_{\\rm aug}$ > 0% is slightly better than $\\rho_{\\rm aug}$ = 0% (especially in early training) in Figure 3’s (b)(f) and is much better in the other 4 tasks. This is because these tasks all have rotation symmetries about z-axis. Note that these tasks include the 3D version of Figure 3’s (a)(c)(e).\n\n**On Weaknesses 5 - limb-based vs joint-based representations**   \nAlthough simulators like MuJoCo often use joint-based representations as default, both representations are convenient to obtain. Given a joint-based representation, the computation of its corresponding limb-based representation is known as *forward kinematics*, which is well-studied in robotics and implemented in simulators such as MuJoCo. We also use MuJoCo’s original APIs for forward kinematics. Therefore, choosing limb-based representations should not limit generalizability.\n\nRegarding justifying limb-based over joint-based representations, it is non-trivial to provide a theory that proves advantages of one over the other (if any), especially in the context of deep RL. That's why we have performed our experiments to give empirical evidence.\n\n**On Question 1**   \nThank you. We will fix it.\n\n**On Question 2**   \nThere is only one rotation applied to a chosen transition (lines 274-277). We also mention rotation in lines 271-273 just to describe how the loss is computed.\nWe will modify the description to make this clear.\n\n**On Question 3**  \nThank you for suggesting the papers as they are indeed related to our work.\nAs these papers are also mentioned by Reviewer X7ZX, please refer to our global response that compares our work with those papers. The comparison clarifies our different and orthogonal contributions.\n\n**On Question 4**  \nThank you for the suggestion. Our data augmentation method can be straightforwardly applied to any off-policy algorithm. We have performed additional experiments applying it to SAC on the tasks of Walker_run and Hopper3D_hop in Figure 1(a)(b) in our rebuttal pdf. The results show our data augmentation has similar effects for SAC.  These results provide evidence that our method is well compatible with other RL algorithms. \n\n**On Limitations**  \nThank you for the suggestion.\nWe would not make assertive claims on real robots, since we did not provide evidence therein.\nHowever, as data is often even more scarce on real robots, we do believe any data augmentation method, including ours in this work, should be valued.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We attach here a pdf that contains additional results and illustrations, which we refer to in our response to individual reviews.

We below compare our work with papers [1-6] suggested by reviwer X7ZX to clarifies our different and orthogonal contribution.
Some of these papers are also suggested by reviewer 5Bxo.

**Our contribution**  
Our contribution is a novel data augmentation transformation, namely ${\rm SO}_{\vec{g}}(3)$ transformation on limb-based state representations, for RL-based robot locomotion. Our experiments on a wide range of simulated locomotion tasks show the advantage of this transformation over alternatives in sample and computation efficiency.

**Comparision with [1-6]**   
Our contribution is different from the papers [1-6] in the following sense:

- [1,2] and our work both consider “dynamics-invariant” data augmentation transformations, but our focus is very different. [1,2] focus on better leveraging known, existing dynamics-invariant transformations, drawing their conclusions mostly from robot navigation and manipulation tasks (e.g., AntMaze, Soccer, panda-gym); while we propose a novel transformation for robot locomotion.

    - Specifically, [1] studies the question of when data augmentation is helpful and concludes the augmentation ratio is crucial (their contribution 3). Their paper’s main body focuses on robot navigation and manipulation, where transformations like goal-relabelling and translations are key to overcoming the sparse reward issue. [1] does use rotation too, in a toy 2D navigation task (Goal2D-v0) where the robot is a particle (no meaningful kinematics) and in the dense-reward MuJoCo locomotion tasks (their Appendix F) where the rotation is performed on joint-based representation and does not yield improvement over no-augmentation baseline (see our response to Weakness 2 for more details). 

    - [2] uses transformations to randomly generate synthetic data and then asks humans to select high-quality ones. Such manual filtering is helpful when doing robot navigation, which [2] focuses on, because humans can easily tell high-quality trajectories based on distance to goal; but it is much more difficult for humans to tell effective joint movements for locomotion. Our method does not require human effort at all. 

- [3] (and their follow-up work MoCoDA [Pitis et al., 2022]) proposes a data augmentation transformation that requires local (causal) independence, so that augmentation can be performed via stitching independent trajectories from decomposed, independent parts, which is useful for tasks like particles moving and 2-arm robot with static base. We focus on locomotion tasks that do not exhibit sparse kinematic interactions between limbs and therefore cannot benefit much from [3]’s method. For example, the cheetah’s two legs are connected through the moveable torso, and therefore we cannot decompose and stitch their separate trajectories.

- [4] focuses on the transformation of reflection that exploits bilateral gait symmetry. Reflections require gait symmetry (e.g., the left leg has the same length, stiffness, etc.), which our MuJoCo tasks do not necessarily have. Some work modifies the MuJoCo robot to enforce gait symmetry before applying reflections (e.g., see Table 1 of [1]). Our rotation transformation instead does not require gait symmetry.  

- [5] also focuses on reflections (they call them "mirror") for locomotion. They perform trajectory-level transformations for on-policy RL, which is technically not sound but they found it "not a critical issue in practice", while we do rotation-based transition-level transformations for off-policy RL. Moreover, their data augmentation does not yield significant improvement over the no-augmentation baseline (see their Figure 3, DUP vs BASE).

- [6] addresses the problem of sparse rewards in reinforcement learning by the goal-relabelling transformation, which is different from our rotation transformation and does not bring much benefit to locomotion tasks.

Meta-review: This paper introduces a novel data augmentation technique for reinforcement learning in continuous control tasks, leveraging euclidean symmetries. The primary contributions include a method to apply Euclidean transformations to state representations, a limb-based state representation enabling effective augmentation, and empirical evaluation on various DeepMind Control Suite tasks.

The reviewers generally agree on the strengths: a novel approach to state-based RL data augmentation, comprehensive experiments, and significant improvements on challenging 3D tasks. However, concerns were raised regarding limited theoretical support, varying effectiveness across tasks, and focus primarily on rotational transformations. There was some disagreement among reviewers regarding the significance of the contribution relative to prior work on dynamics-invariant augmentations and the overall effectiveness of the method given its varied performance across tasks. 

Despite these limitations, the paper presents a promising approach with some notable results on complex tasks. The authors have adequately addressed most concerns during the discussion phase. Therefore, I recommend accepting this paper for a poster presentation.
Decision: Accept (poster)