Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Control

Authors: jinzhu  luo
dingyang  chen
qi  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 04:50:45

Abstract: Data augmentation creates new data points by transforming the original ones for an reinforcement learning (RL) agent to learn from, which has been shown to be effective for the objective of improving data efficiency of RL for continuous control. Prior work towards this objective has been largely restricted to perturbation-based data augmentation where new data points are created by perturbing the original ones,
which has been impressively effective for tasks where the RL agent observe control states as images with perturbations including random cropping, shifting, etc. This work focuses on state-based control, where the RL agent can directly observe raw kinematic and task features, and considers an alternative data augmentation applied to these features based on Euclidean symmetries under transformations like rotations. We show that the default state features used in exiting benchmark tasks that are based on joint configurations are not amenable to Euclidean transformations. We therefore advocate using state features based on configurations of the limbs (i.e., rigid bodies connected by joints) that instead provides rich augmented data under Euclidean transformations. With minimal hyperparameter tuning, we show this new Euclidean data augmentation strategy significantly improve both data efficiency and asymptotic performance of RL on a wide range of continuous control tasks.

Main paper: 
 introduction: 
 While reinforcement learning (RL) has enjoyed impressive success on continuous control problems, especially when empowered by expressive function approximators like deep neural networks, improving its notoriously poor data efficiency remains challenging. Recently, exploiting the idea of data augmentation, the RL community has made significant progress on improving data efficiency as well as the asymptotic performance of RL for continuous control with the agent observing images as the state representations [1, 2, 3, 4] . CURL [1] utilized data augmentation for learning contrastive representations [5, 6] out of their image encoder, as any auxiliary learning objective in the RL setting. Other works, such as DrQ [3, 4] and RAD [2] , directly used image data augmentations for RL without any auxiliary objectives. Despite these technical differences in how data augmentation is integrated into RL, these works share the key procedure of perturbation-based image augmentation: to create new data for learning, the image in a state representation is transformed by applying perturbations such as random crop, random shift, color jitter, etc. Instead of image-based continuous control, this paper focuses on the more primitive problem of statebased continuous control, where the agent can observe raw physical quantities such as position and velocity as the features to represent the underlying world state. Like the image-based case, existing work on state-based data augmentation has been relying on introducing perturbations to original state features, examples including adding Gaussian noise and scaling the features with random amplitudes [2] . However, unlike the image-based case, the perturbed data for state-based continuous control only leads to limited improvements, sometimes even negative effects, on the data efficiency of RL, as found in existing work [2] and our experiments in Section 5. Such ineffectiveness is due to the fundamental difference between image-based and state-based features: from a control-theoretic perspective, the perturbed physical quantities as state-based features are largely uncorrelated with the original ones, in the sense that although the original transition comes from the ground truth dynamics and reward functions, the perturbed transition does not necessarily; this is in contrastive difference from images, where perturbations such as shifting create new views of the same underlying world state.To overcome the limitations of perturbation-based data augmentation, this paper advocates the alternative of Euclidean data augmentation for state-based continuous control. The key idea is that, because our control tasks operate in the 2D/3D space, they exhibit Euclidean symmetries: transformations such as rotation and translation leave the transition dynamics and reward function invariant. Due to the invariancy, these symmetry-preserving Euclidean transformations are ideal operations for data augmentation, because the transformed data will be guaranteed to be valid samples from the task's dynamics and reward.While there exists prior work on Euclidean symmetries in RL [7, 8, 9, 10] , the idea of exploiting them as a data augmentation method for state-based continuous control is surprisingly underexplored. This is because current benchmarks [11, 12] by default use the joint configurations of the robotlike agent as its state features, which are mainly angular quantities that are invariant to Euclidean transformations and therefore not amenable to data augmentation, as we will explain in detail in Section 4. To overcome this, we innovatively use configurations of the limbs, i.e., the rigid body links connected by the joints, as an equivalent state representation to replace the default joint-based one. Because limb configurations are specified by physical quantities such as linear position and velocity, they are equivariant under Euclidean transformations and therefore provide rich augmented data.Our algorithmic innovations based on the introduced ideas lead to significant improvement in data efficiency and asymptotic performance of RL for state-based continuous control. Building from DDPG [13] as the base RL algorithm, our limb-based state representation alone improves the performance on most tasks from the DeepMind Control Suite [12] , a standard continuous control benchmark for RL, and additional Euclidean data augmentation is necessary to obtain the best performance for almost all tasks, especially for the ones that have large degrees of freedom and are historically hard to solve. To name a few hardest tasks, on the Humanoid_run task, standard DDPG achieves an episode reward below 100, while our method achieves 150, both after 5M timesteps; on Hopper3D_hop, standard DDPG achieves an episode reward below 40, while our method achieves more than 200, both after 2M timesteps. 
related work: 
 Data augmentation for image-based continuous control. Existing works in data augmentation for RL mostly have focused on the online setting for image-based continuous control tasks. This is because it is relatively straightforward to obtain augmented images (e.g., through random cropping/shifting), which enables learning representations of the high-dimensional image input that facilitates optimizing the task reward. In this line of work, CURL [1] utilizes contrastive representation learning to the image representations, jointly with the RL objective of reward optimization; RAD [2] and DrQ [3, 4] use the augmented data directly for RL without any auxiliary objective. SVEA [14] focuses on improving stability and sample efficiency under image-based data augmentation by mitigating the high-variance Q-targets. CoIT [15] works on maintaining the invariant information unchanged under image-based data augmentation. Different tasks often benefit from different types of data augmentation, but the manual selection is not scalable. The work of [16] automatically applies the types of data augmentations that are most suitable for the specific tasks.Data augmentation for state-based continuous control. This paper focuses on data augmentation for state-based continuous control, which only a few prior works have explored. Specifically, RAD explores the augmentation of injecting noise to state variables through additive Gaussian or random amplitude scaling [2] . Different from these unprincipled, perturbation-based augmentation, this paper advocates a principled augmentation method through Euclidean symmetries for state-based continuous control. Corrado et al. [17, 18] also consider principled data augmentation transformations that are not perturbation-based but focus on better leveraging existing transformations, drawing their conclusions mostly from robot navigation and manipulation tasks, while we propose a novel transformation for robot locomotion. Pitis et al. [19, 20] propose a data augmentation transformation that requires local (causal) independence, so that augmentation can be performed via stitching independent trajectories from decomposed, independent parts, which is useful for tasks such as particles moving and two-arm robots with a static base. We instead focus on locomotion tasks that do not exhibit sparse kinematic interactions between limbs and therefore cannot benefit much from their method. For example, the legs of a cheetah-like robot are connected through the moveable torso, which we cannot decompose and stitch their separate trajectories.Euclidean symmetries in RL. Symmetries in RL refer to the structural property of invariant transition dynamics and reward functions under certain transformations, leading to the existence of invariant optimal values and policies [7, 8] . Euclidean symmetries thus correspond to the transformations being translations, rotations, and/or reflections, which are applicable when the state/action space is Euclidean. Most prior works focus on image-based control problems, e.g., Atari games and robot manipulation from image observations [9] , or gridworld-like domains exhibiting only discrete symmetries (e.g., 4-way Cartesian rotation) [8, 21] . A closely related work by Abdolhosseini et al. [22] uses reflection-based data augmentation (they call them "mirror") for locomotion. They perform trajectory-level transformations for on-policy RL, which is technically not sound and does not yield significant improvement over the no-augmentation baseline (see Figure 3 therein ). Instead of relying on data augmentation, some of these works employ equivariant neural networks as their agent architectures to exploit the symmetries [8, 9] , which often incurs significantly higher computational cost. The reason that existing works largely overlook data augmentation based on Euclidean symmetries in state-based continuous control is two-fold: 1) most standard benchmark tasks operate in the 2D space [11, 12] , and these planar tasks exhibit limited Euclidean symmetries compared to the 3D counterparts; and 2) the benchmark tasks employ the state representations based on the joints' configurations and velocities, which in nature exhibit very limited Euclidean symmetries. This paper addresses these issues to unlock the potential of data augmentation through Euclidean symmetries for state-based control via RL. 
online rl for state-based continuous control: 
 We formulate a state-based continuous control task (or, simply, task) as a Markov decision process (MDP), defined as a tuple (S, A, p, r, γ). At each discrete time step t, the agent fully observes the state in the continuous state space, s t ∈ S ⊂ R d , and chooses an action in the continuous action space to take, a t ∈ A ⊂ R m ; the next state is sampled from the transition dynamics p with conditional probability density p(s t+1 |s t , a t ), while the reward function r : S × A → R yields a scalar feedback r t := r(s t , a t ). The agent uses a policy π : S → ∆(A) to choose actions where ∆(A) is the collection of probability measures on A, yielding a trajectory of states, actions, and rewards, (s 0 , a 0 , r 0 , s 1 , . . .), where the initial state s 0 is sample from an initial state distribution with density d 0 and a t ∼ π(•|s t ) Without prior knowledge of p and r, the reinforcement learning (RL) agent's goal is to obtain a policy from its experience (i.e., yielded trajectories) which maximizes the expected cumulative discounted reward, E π [ ∞ t=0 γ t r t ] where γ ∈ [0, 1). The online RL setting assumes the agent has the ability to interact through the MDP to obtain an increasing amount of trajectories while refining its policy.Deep Deterministic Policy Gradient (DDPG) [13] is a state-of-the-art online RL algorithm for continuous control, which parameterizes and learns a critic Q θ : S × A → R and a deterministic policy π ϕ : S → A. The critic is trained by minimizing the one-step temporal difference (TD) error L(θ) = E (st,at,rt,st+1)∼D online [(Q θ (s t , a t )-r t -γQθ(s t+1 , π ϕ (s t+1 )) 2 ], where D online is the replay buffer storing past trajectories and θ is an exponential moving average of the critic parameter, referred to as the target critic. Concurrently, the policy is trained by employing Deterministic Policy Gradient (DPG) [23] and minimizing L(ϕ) = E st∼D online [-Q θ (s t , π ϕ (s t ))], so that π ϕ (s t ) approximates arg max a Q θ (s t , a). The policy, after adding noise for exploration, is used as a data-collection policy to generate and store new trajectories in replay buffer D online , which is interleaved with the critic and the actor parameter update steps.Perturbation-based data augmentation. Data augmentation generates new, artificial data through transformations of original data. Better data efficiency can often be achieved by learning from both the original and augmented data. For mastering continuous control tasks via RL, prior works perform data augmentation by perturbation, through 1) transformations of adding noise to original data, examples including performing random image cropping, color jittering, translations, and rotations when states can only be observed and represented as images (i.e., image-based continuous control) [24, 25, 1, 2, 3, 4] and 2) transformations of adding Gaussian noise and random amplitude scaling when states (e.g., positions and velocities) can be directly observed by the agent [2] .Symmetry-based data augmentation. This work focuses on state-based continuous control and departs from prior works by performing data augmentation through MDP symmetries [7, 8] . Symmetries in an MDP, if existing, can be described by a set of transformations on the state-action space indexed by g ∈ G, where such a transformation leaves the transition dynamics and reward function invariant. Formally, we define a state transformation and a state-dependent action transformation as T S g : S → S and T A g,s : A → A, respectively, and the invariance is expressed accordingly as, for all g ∈ G, s, s ′ ∈ S, a ∈ A,EQUATIONThe invariance naturally makes these transformations sound and effective for data augmentation purposes, especially in continuous control tasks where transition dynamics is often (near-)deterministic.Next, we detail our data augmentation method based on Euclidean symmetries for continuous control.4 Our method: Euclidean data augmentation for continuous control 
kinematics of our continuous control tasks: 
 We consider the continuous control problem for an agent (e.g., a robot) consisting of n rigid bodies, referred to as limbs, that are connected by joints to form a certain morphology and can move in the 2D planar space or the 3D space. Examples of such agents include 2D Cheetah and 2D Hopper from DeepMind Control Suite (DMControl) [12] as well as their 3D variants [26] , which will be presented in our experiments. Without loss of generality, we present our method in the 3D setting.Tree-structured morphology and actuation. The connected limbs form a certain morphology that can be represented as an undirected graph, where each node i ∈ {1, • • • , n} represents a limb and an undirected edge (i, j) exists if there is a joint connecting nodes/limbs i and j. We focus on the case where the graph is connected and acyclic to become a tree, where the root node, indexed as i = 1, is often the torso or the base in the agent's morphology. Any two connected limbs are rigidly attached to each other via their joint to provide degrees of freedom (DoFs) between the two limbs. A joint provides either 1-, 2-, or 3-DoF rotation for the child node/limb relative to its parent: in general, as a 3D rigid body, the child can rotate about yaw-, pitch-, and/or roll-axes relative to its parent, creating 3 DoFs; for a 2D planer task, the agent operates in the xz-plane and the joints therein can only rotate about the y-axis (pitch), creating only 1 DoF. The joints are actuated to produce a scalar torque for each of these rotation DoFs. We let d i ∈ {1, 2, 3} denote the number of DoFs of the joint with node i > 1 as its child, and the collection of d i scalar torques is denote as a i ∈ R di , resulting in the action for the agent as a = (a i ) i>1 ∈ R m where m = i>1 d i . The root node, as a 3D rigid body, has up to 6 DoFs relative to the global frame, and we let d 1 ≤ 6 denote the number of its DoFs.As the action of torques specifies acceleration, a Markovian state of such an agent has to specify its configuration, i.e., how the limbs occupy the space and how the joints are rotated, as well as its velocity, i.e., the time derivative of its configuration. We refer to these physical quantities as the agent's kinematic features and enumerate here several of them that will be closely related to our discussion: • q i , qi ∈ R di respectively denote the configuration and the velocity (i.e., the configuration's time derivative) of the joint with limb i as its child. For i > 1, we have d i ≤ 3 corresponding to each active joint rotation axis, and q i consists of the rotation angles. For i = 1, there is no joint with limb i = 1 (i.e., torso) as its child; in this case, q 1 ∈ R d1 specifies the configuration corresponding to all d 1 ≤ 6 DoFs of the limb.• ⃗ r i,1 , • • • , ⃗ r i,di ∈ R 3denote the d i rotation axes of the joint with limb i > 1 as its child, relative to the global frame.• ⃗ p i , ⃗ v i ∈ R 3 respectively denote the (translational) position and linear velocity (i.e., the position's time derivative) of limb i ≥ 1, relative to the global frame.• ⃗ R i ∈ R 3×3 and ⃗ ω i ∈ R 3respectively denote the Cartesian orientation and angular velocity (i.e., the orientation's time derivative) of limb i ≥ 1, relative to the global frame.Task features. Our continuous control tasks are mostly of locomotion, i.e., the agent is tasked to move (part of) itself from one place to another. In tasks where the agent is tasked to move in a certain direction (e.g., Cheetah), we let ⃗ t ∈ R 3 denote the unit vector in that direction. In tasks where the agent aims to reach a target point ⃗ p * ∈ R 3 with one of its limb i (e.g., Reacher where i = n = 3 being the last limb), we let ⃗ t = ⃗ p * -⃗ p i ∈ R 3 denote the vector from the limb to the target point.Example: 2D/3D Cheetah. The original Cheetah from DMControl operates in the xz-plane [12] with the target direction being ⃗ t = [1 0 0], i.e., the agent is tasked to move along the positive x-axis.! " # * ! " # * ) & ∈ SO ' 3 / / Figure 2: SO ⃗ g (3) rotation in Cheetah.Its torso is the root node and has d 1 = 3 DoFs: moving along the x-axis with position p x 1 ∈ R, moving along the z-axis with position p z 1 ∈ R, and rotating about the y-axis with rotation angle β y 1 ∈ [0, 2π), and correspondingly we haveq 1 = (p x 1 , p z 1 , β y 1 ) ∈ R 3 .The other 6 limbs (front/back thigh, shin, and foot) form the front and back legs attached to the torso, forming the morphology tree as shown in Figure 1 . For the 2D planar case, the 6 joints can only rotate about the y-axis, and therefore, for i ∈ {2, • • • , 7}, we haved i = 1, q i = (β y i ) ∈ R 1 with β yi being the rotation angle, and its only rotation axis ⃗ r i,1 is fixed along the y-axis. In our experiments, we extend this 2D Cheetah to its 3D variant, where the torso has all d 1 = 6 DoFs to freely move the in 3D space and each of the 6 joints, i ∈ {2, • • • , 7}, has d i = 3 DoFs to be able to freely rotate about the yaw-, pitch-, and roll-axes (⃗ r i,1 , ⃗ r i,2 , ⃗ r i,3 ).If the agent is placed in the free space (free of obstacles and external forces), its transition dynamics and reward function are invariant to the transformations including translations, rotations, and reflections. Formally, these transformations together form the Euclidean group of G = E(3) in Equation ( 1), where the state should include the agent's kinematic features (i.e., configuration and velocity) and the task features. This is the basis of our method of Euclidean data augmentation.For the tasks in our experiments, the agent is not placed in the free space and, instead, is placed on the level surface of xy-plane (i.e., the ground) and subject to the external force of gravity ⃗ g ∈ R 3 towards the negative z --axis. Therefore, the task is not invariant to arbitrary rotations/reflections but only to rotations about the direction of ⃗ g (or equivalently, the z-axis), which subsumes reflections along this direction. For the reasons mentioned above, our Euclidean data augmentation only includes rotations about ⃗ g, which form a subgroup of SO(3) (which includes all 3D rotations) and is denoted with its matrix representation asSO ⃗ g (3) = {R ∈ R 3×3 | R ⊤ R = I, det R = 1, R⃗ g = ⃗ g}.Alternatively, the matrices can be parameterized by the angle rotated about z-axis (i.e., yaw) as α ∈ [0, 2π) asSO ⃗ g (3) = {R α,β,γ ∈ R 3×3 | α ∈ [0, 2π), β = γ = 0)} where α, β, and γ are yaw, pitch, and roll angles, respectively. We will abbreviate the matrix as R α since β = γ = 0.Note that other Euclidean transformations, i.e., translations or reflections, are not available in our tasks: As explained later, the agents in the tasks use egocentric representation and therefore it's already translation-invariant. Reflections require gait symmetry (e.g., the left leg has the same length, stiffness, etc.), which the agents in the tasks do not necessarily have. Some prior work has modified the agent to enforce gait symmetry before applying reflections (e.g., Corrado and Hanna [18] ), but we do not.For a state rotated by R α , the kinematic features and task features are accordingly transformed in the following three ways (see Figure 2 for an illustration in Cheetah):• Equivariant features: We have denoted these features with arrow as 3D vector ⃗ x ∈ R 3 which will be rotated intoR α ⃗ x ∈ R 3 , including ⃗ r i,1 , • • • , ⃗ r i,di , ⃗ p i , ⃗ v i , columns of ⃗ R i , ⃗ ω i , and ⃗ t i .• Invariant features: For the joint with a non-torso limb i > 1 as its child, its configuration q i specifies the rotation angles to the active rotation axes. Therefore, {q i , qi } i>1 and torque action a will stay invariant.• Transformation of q 1 and q1 : Note that elements of the torso's position ⃗ p 1 are also part of q 1 . This part of q 1 will be transformed accordingly. The rest of q 1 includes β z 1 , β y 1 , and/or β x 1 , the yaw, pitch, and/or roll angles of the torso. Since rotation R α is a yaw-only rotation, we have β z 1 → β z 1 + α while β y 1 and β x 1 will stay invariant. Similar arguments apply to q1 .We below present a few implementation details of SO ⃗ g (3)-data augmentation in our experiments.Limb-based kinematic state representation. Besides the task features, the Markovian state should include, for the agent itself, kinematics features that are sufficient to represent its configuration and velocity. Not all kinematic features introduced in Section 4.1 are independent and therefore a sufficient kinematic state representation needs only to include a subset of them. Current benchmarks for RL-based continuous control (e.g., DMControl [12] and Gym-MuJoCo [11] ) all employ the joint-based kinematic state representation, which includes only joint configurations and velocities {q i , qi } i≥1 , which is indeed a valid kinematic state representation because all limbs are rigid bodies. However, as we have discussed, this joint-based kinematic state representation is not amenable to SO ⃗ g (3)-data augmentation, since most of its features are invariant (only q 1 and q1 are not invariant).To provide rich augmented data under SO ⃗ g (3), we use an alternative kinematic state representation that is mostly based on limb features, including ⃗R 1 , ⃗ ω 1 , {⃗ p i , ⃗ v i } i≥1 , and {⃗ r i,1 , • • • , ⃗ r i,di } i>1 . For torso i = 1, its configuration and velocity are included in { ⃗ R 1 , ⃗ ω 1 , ⃗ p 1 , ⃗ v 1 }.For limb i > 1, its translational position and velocity are included in {⃗ p i , ⃗ v i }; its orientation is included in its rotation axes {⃗ r i,1 , • • • , ⃗ r i,di }, so we do not further include ⃗ R i ; and its angular velocity can be recovered from its rotation axes and its velocity, so we do not further include ⃗ ω i .The kinematic features listed above are both sufficient and necessary in general for an agent operating in the 3D space. For a 1-DoF joint (d i = 1) that is typical in 2D planar tasks, the orientation of the torso, ⃗ R 1 , is sufficient to determine the orientation of its active rotation axis. Therefore, we exclude the rotation axes for all the 1-DoF joints and include only for the 2-and 3-DoF joints, resulting in the final set of features in our limb-based kinematic state representation:EQUATIONEgocentric kinematic features. In these benchmark continuous control tasks, the agent can only observe its egocentric position, that is, it observes the position of its torso as the origin and the positions of other limbs and the target relative to its torso. For example, when using the joint-based kinematic state representation for 2D Cheetah, the configuration of torso q 1 = (p x 1 , p z 1 , β y 1 ) ∈ R 3 is observed as (0, p z 1 , β y 1 ). Consistently, our limb-based representation subtracts the torso's translational position for every limb, i.e., ⃗ p i → ⃗ p i -⃗ p 1 for all limbs i ≥ 1.Additional sensory observations. Besides the kinematic and task features, the agent has additional sensory observations in its original state representation that can be classified as either equivariant or invariant features under SO ⃗ g (3) rotations. For example, Hopper from DMControl has two sensors for detecting the touch of its two feet, each yielding an invariant scalar feature. Refer to Table 2 in Appendix A.1 for a complete list of sensory observations for all tasks. Our complete state representation includes these sensory inputs, along with the kinematic and task features, to ensure equivalence to the original state representation.Our data augmentation method can be applied to any off-policy RL algorithm, and our empirical study applies it to DDPG as follows. After sampling a batch of B transitions of the form (s t , a t , r t , s t+1 ) from online replay buffer D online , we randomly choose a subset of B aug (≤ B) transitions and perform random rotations R α ∈ SO ⃗ g (3) to them, with α randomly chosen for each of the B aug transitions. B aug is the only additional hyperparameter that our method introduces on top of the original DDPG. Specifically, we transform states s t and s t+1 as described in Section 4.2, keep action a t and reward r t invariant, and use the transformed transition to compute the DDPG losses (cf. Section 3.2). 
experiments: 
 Environment and tasks. All the tasks in our experiments are provided by the DeepMind Control Suite (DMControl) [12] powered by the physics simulator of MuJoCo [27] , which has become a common benchmark for RL-based continuous control. Specifically, we include 7 tasks originally from DMControl: Cheetah_run, Hopper_hop, Walker_run, Quadruped_run, Reacher_hard, Humanoid_run, Humanoid_stand, and our modified Cheetah3D_run, Hopper3D_hop, and Walker3D_run, which are the 3D variants of their original 2D planar counterparts. The original tasks involving Quadruped and Humanoid are already 3D, while it is not straightforward to extend Reacher_hard to a 3D variant. This results in a total number of 10 tasks. For all tasks in DMControl, an episode corresponds to 1000 steps, where a per-step reward is in the unit interval, i.e., r t ∈ [0, 1]. The original tasks in DMControl employ the joint-based kinematic state representation. We use MuJoCo's built-in data structures and functions (e.g., mjData.xpos for the translational position, mjData.object_velocity for linear and angular velocity [28] ) to obtain the kinematics features needed for our limb-based representation. Refer to Appendix A.1 for a detailed description of the state features for all the tasks.Baselines. We build our data augmentation on top of DDPG, following the implementation in [10] that has achieved state-of-the-art performance on the chosen tasks (comparable or better than, e.g., TD3 [29] , Soft Actor-Critic (SAC) [30] , e.g., refer to [10] for a comparison). Refer to Appendix A.2 for a full list of the DDPG's hyperparameters. For reference, we also run and provide results of standard SAC. Our method introduces only one additional hyperparameter, B aug , the number of transitions to be transformed in a batch of B transitions. We perform a hyperparameter search over B aug /B =: ρ aug ∈ {0, 25%, 50%, 75%, 100%} separately for each task, while keeping all other hyperparameters the same as DDPG default. We compare our SO ⃗ g (3)-data augmentation with two alternatives considered in the prior work of [2] , both operating on the original joint-based kinematic state representation: The Gaussian noise (GN) method adds a standard Gaussian random variable to the state vector, i.e., s → s + z where z ∼ N (0, I); and random amplitude scaling (RAS) multiplies the uniform random variable to the state vector element-wise, i.e., s → s ⊙ z with ⊙ denoting element-wise product, where z ∼ Uni[α, β] with α = 0.5, β = 1.0 as chosen in the prior work of [2] . We additionally compare to using equivariant neural networks, specifically SEGNN [31] , to instantiate the agent's policy and critic, as an alternative method to exploit the Euclidean symmetries. Refer to Appendix A.3 for the details of our SEGNN implementation. Training details. Each training run consumes 2M time steps (1000 steps per episode for 2000 episodes) for all the tasks except for Humanoid_run and Humanoid_stand, for which each training run consumes 5M steps. Evaluation is performed every 10000 steps by averaging the episodic rewards of 10 episodes, which is reported in our figures as the y-axis. In all the figures, we plot the mean performance over 5 seeds together with the shaded regions which represent 95% confidence intervals.The training runs are computed by NVIDIA V100 single-GPU machines, each taking roughly 2 hours to finish 1M training time steps for our method and all the baselines except for the SEGNN baseline, which takes roughly 70 hours to finish 1M steps.As SEGNN is prohibitively expensive in computation, we only run it on the task of Reacher_hard, the smallest one among all 10 tasks, and defer the results to Section 5.2. Figure 3 presents the learning curves to compare our method against all other baselines on the rest 9 tasks. For our method, we present both ρ aug = 0% and the task-specific best positive ρ aug ∈ {25%, 50%, 75%, 100%} to separate the effects of our limb-based kinematic state representation and the data augmentation on top of it. We make the following observations from the results:(i) DDPG is comparable to or significantly better than SAC in all 9 tasks except for Cheetah_run and Humanoid_run, which justifies our choice of DDPG as the base RL algorithm.(ii) As existing alternative data augmentation methods for state-based continuous control, the baselines of GN and RAS do not offer significant improvement in data efficiency. They even introduce negative effects on performance on most tasks, except for Cheetah3D_run and Humanoid_stand where RAS improves data efficiency over its based algorithm of DDPG.(iii) With ρ aug = 0%, i.e., with the limb-based state representation alone, our method improves data efficiency over DDPG on the 6 tasks shown in the first 2 rows in Figure 3 , and in 5 out of the 6 tasks (excluding Hopper3D_hop), ρ aug = 0% is comparable to the best ρ aug > 0%. On the rest of 4 tasks (Hopper3D_hop and 3 tasks in the last row), a positive ρ aug > 0 is crucial to attain the best performance among all the methods. To summarize, our method reliably improves DDPG, the state-of-the-art RL algorithms on the continuous control tasks. The improvements are remarkably significant, especially on tasks with rich 3D DoFs and/or large numbers of state features: for example, the improvements on Hop-per3D_run are more salient on its 2D variant Hopper_run; our aggressive data augmentation (ρ aug = 100%) is necessary to achieve the best performance on Humanoid_run and Humanoid_stand, the hardest two tasks where no baseline is able to learn on both.Employing equivariant neural networks such as SEGNN as the agent policy and critic architecture is an alternative method to exploit the Euclidean symmetry but at the same time more computationally expensive than data augmentation. As we are not able to afford to finish the SEGNN baseline for all tasks, we have run it only for Reacher_hard, the smallest one by the number of limbs/joints. Figure 4 reports the learning curves of our method, SEGNN, and all other baselines on Reacher_hard, with the x-axis being time steps and hours to compare data and computational efficiency, respectively. As the results show, SEGNN is the most data efficient, followed by our data augmentation method (ρ aug = 25%), both clearly outperforming the others. However, our method introduces minimal computation in addition to its DDPG base algorithm, both taking roughly 2 hours to finish 1M steps after convergence, while SEGNN takes more than 70 hours to finish 1M steps and more than 10 hours to converge.In Sections 5.1 and 5.2, we have presented the learning curves of our method with ρ aug = 100% and best task-specific ρ aug > 0%). Here, we present further results detailing the effect of ρ aug , with the learning curves of all values of ρ aug on all tasks given in Appendix B.1. We observe that: 1) On relatively simple tasks with 2D planer DoFs and small numbers of limbs/joints, ρ aug > 0% does not provide significant improvements over ρ aug = 0% examples including Cheetah_run as shown in Figure 5a ; 2) Data augmentation (ρ aug > 0%) is crucial to best performance on hard tasks with 3D DoFs and large numbers of limbs/joints, examples including Hopper3D_hop and Humanoid_run in Figures 5b and 5c , respectively; and 3) The best ρ aug is task-specific and not necessarily the largest possible ρ aug = 100%. For example, ρ aug = 100% hinders performance on Hopper3D_hop but is the best on Humanoid_run. These observations underscore the task-specific nature of augmentation strategies, which aligns with findings from previous studies in the field of computer vision, where the idea of data augmentation originated and has prevailed. Cubuk et al. [32] demonstrated that different datasets such as CIFAR-10, SVHN, and ImageNet require distinct augmentation strategies. Similarly, Zoph et al. [33] showed significant differences in the optimal augmentation strategies for various object detection tasks, highlighting the task-specific requirements of augmentation policies. Ho et al. [34] further emphasized this point in their study on Population Based Augmentation, demonstrating that the best augmentation policies vary across tasks, underscoring the need to tailor augmentation strategies to specific tasks and datasets.Indeed, one can perform SO ⃗ g (3) rotations on the original joint-based state representation. In the presented tasks, only the torso (i = 1) has features for orientation (in its up to 6 DoFs) while features of other limbs (i > 1) are just angles and angular velocities of the hinges. Therefore, under a rotation, only the torso's orientation features are changed, while other features stay unchanged. Therefore, we hypothesize this data augmentation by rotating original joint-based features would bring little benefit to the tasks. This is because torso features make up only a fraction of all features when the number of limbs is large (which is the case in these tasks). Our limb-based representation instead rotates all limbs to provide richer augmentation. The hypothesis is supported by results of prior work (e.g., see Figure 16 in Corrado and Hanna [18] ). We here also conduct a comparison, with results in Figure 6 showing that joint-based data-augmentation is less efficient than our limb-based method. 
conclusion: 
 We have motivated, formalized, and developed the idea and method of a new data augmentation method that aims to improve the performance of RL, as measured by its data efficiency and asymptotic reward, for state-based continuous control. The key idea of our data augmentation method is to create additional training data by performing Euclidean transformations on the original states, which is justified by the Euclidean symmetries inherent in the continuous control tasks. To make the states more amenable to Euclidean transformations, we turn to a state representation based on limbs' kinematic features, rather than on joints' configurations as done by default in prior works. Our new method significantly improves the performance for a wide range of state-based continuous control tasks, especially for tasks with rich 3D motions and large numbers of limbs and joints.Limitations. Our work focuses on robot locomotion tasks as instances of continuous control, which clearly exhibit Euclidean symmetries. Other robotics tasks (e.g., navigation) and many applications that operate in the 3D physical space should also exhibit Euclidean symmetries. There are indeed continuous control tasks that might not exhibit Euclidean symmetries, such as those in electrical and power engineering, which we do not consider. We observe two limitations that would inspire future work: 1) To obtain the best performance, our data augmentation method needs task-specific turning of its hyperparameter, ρ aug , that controls the proportion of data to be transformed for learning. Future work in this direction is needed to ease task-specific turning, either by adaptive tuning strategies [35] or training techniques to make learning more robust and less sensitive to the hyperparameter [36] ;2) The proposed method requires knowledge and annotation of strict Euclidean symmetries in the continuous tasks, which might not be easily available especially when the external forces are noisy and hard to detect, e.g., including wind conditions in the wild field along with gravity. This prompts future work of automatic discovery and exploitation of irregular, approximate Euclidean symmetries.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The paper include enough details necessary to understand the results in Section 5 and Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: As described in Section 5, the paper reports error bars corresponding to 95% CI.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: That the paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The paper properly credits the original creators as cited in Section 5. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide well-documented code, as our new asset, in the supplemental material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Task details Kinematic features. Table 1 lists for all the 10 tasks the kinematic parameters including number of limbs n, number of DoFs of the torso limb d 1 , numbers of 1-2-, and 3-DoF joints n 1-DoF , n 2-DoF , and n 3-DoF . There are three possible values for d 1 :• d 1 = 0. The torso is fixed, e.g., in Reacher_hard.• d 1 = 3. The torso has all 3 DoFs: moving along the x-axis, moving along the z-axis, and rotating about the y-axis, which is the case in 2D planar tasks.• d 1 = 6. The torso has all 6 DoFs, which is the case in 3D tasks.These parameters are sufficient to recover other kinematic parameters of a task. For example, action size m = i>1 d i = n 1-DoF + 2n 1-DoF + 3n 1-DoF ; according to (2), the size of our limb-based kinematic features of a task should be 3× 3 + 3 + 3n + 3(2n 1-DoF + 3n 1-DoF ).A caveat is that, for the number of limbs n in Table 1 , we do not count the bodies that do not have any DoF with respect to its parent, examples including the finger with respect to wrist in Reacher_hard. An exception is the torso in Reacher_hard, where we do count it in n. Such bodies are welded to its parent. By doing so, we keep the relationship of n = n 1-DoF + n 2-DoF + n 3-DoF + 1 in Table 1 . In our implementations, however, we do include the kinematic features of these bodies.Table 1 : Kinematic parameters of the tasks. n: number of limbs. Sensory observations. Table 2 lists the sensory observations for all the 10 tasks.Table 2 : Sensory observations in the tasks. Reacher_hard: The task has rotation-equivariancy in the xy plane, and no translation-invariancy due to the hinge fixed at the origin. The state contains The state contains the positions of the target, finger, hand, arm, and root (fixed at origin), and the velocities of the finger, hand, arm, and root (fixed constant 0). All these features are equivariant and they are transformed into the state-action based and state based Euclidean graphs for critic and actor, respectively. Specifically, the point set in the point cloud is V = {target, finger}. The node feature for point finger : f node,finger feature = {[v j , ∥v j ∥]} j∈{finger,hand,arm,root} ∪ {[x j , ∥x j ∥]} j∈{hand,arm,root} } ∪ {a, ∥a∥}. The node feature f node,target feature for point target is {0}. The node attribute contains the node_type: ∀i ∈ V, f node attribute = [node_type(i)]. The Observation based Euclidean graph is the same as the state-action based Euclidean graph, except that action is not included in the node feature. Justification: We provide our code in the supplemental material along with instructions to reproduce the main experimental results, which will be open-sourced upon the acceptance of the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: There is no societal impact of the work performed, as the paper addresses the learning problem in simulated continuous control tasks, which does not raise immediate societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://github.com/JinzhuLuo/EuclideanDA 38th Conference on Neural Information Processing Systems (NeurIPS2024). 


Decision: This paper introduces a novel data augmentation technique for reinforcement learning in continuous control tasks, leveraging euclidean symmetries. The primary contributions include a method to apply Euclidean transformations to state representations, a limb-based state representation enabling effective augmentation, and empirical evaluation on various DeepMind Control Suite tasks.

The reviewers generally agree on the strengths: a novel approach to state-based RL data augmentation, comprehensive experiments, and significant improvements on challenging 3D tasks. However, concerns were raised regarding limited theoretical support, varying effectiveness across tasks, and focus primarily on rotational transformations. There was some disagreement among reviewers regarding the significance of the contribution relative to prior work on dynamics-invariant augmentations and the overall effectiveness of the method given its varied performance across tasks. 

Despite these limitations, the paper presents a promising approach with some notable results on complex tasks. The authors have adequately addressed most concerns during the discussion phase. Therefore, I recommend accepting this paper for a poster presentation.

Reviews: review: paper_summary: paper_summary: This work proposes a novel data augmentation approach that leverages Euclidean symmetry for continuous control to improve the efficiency and performance of RL algorithms. The authors integrated their approach with DDPG and performed a series of comparison against the vanilla SAC algorithm, other augmentation techniques, and other equivariant methods. The results show some improvements in performance across a range of continuous control tasks, especially those with rich 3D motions and large number of limbs and joints.
strength_weakness: strength_weakness: 1. This method is primarily applicable to tasks with clear Euclidean symmetries. So, it seems that in environments where such symmetries are not evident or relevant, the approach may not be applicable or effective. While the 'Limitation' section briefly touches upon this issue, it is not clear what categories of continuous control problems this approach will be applicable to.  
2. The effectiveness of Euclidean transformations may vary depending on whether the task is set in a 2D or 3D space, with potentially limited benefits in more constrained settings (like planar movements).
3. As the 'Limitation' section suggests, practical application of the framework will be difficult due to the task-specific hyperparameter tuning requirements. While the author suggests adaptive hyperparameter tuning as a potential future work, some insights into what characteristics of the environment dynamics dictate the optimal choice of hyperparameters such as \rho_{aug} would be helpful. Also, in the context of practical application, switching from joint-based to limb-based configurations complicate the state representation and requires significant changes to the training environment.
questions: questions: 1. In Section 5.1, the authors say "To summarize, our method reliably improves DDPG and SAC, the state-of-the-art RL algorithms on the continuous control tasks." However, it looks like only vanilla SAC is used in this work for comparison. Hence, the mention of improving SAC should be removed.

2. In line 194 p^y_1 should be  p^z_1.
limitations: limitations: Technical limitations are highlighted in the paper.
paper_summary: paper_summary: This paper proposes a data augmentation technique that leverages Euclidean symmetries (e.g. rotational symmetry) in a task's dynamics to generate augmented data. When a task's state features do not have such symmetries, the paper also discusses how to define a new state representation with these symmetries (so that data augmentation can be applied). Empirically, the paper shows that RL is more data efficient with this data augmentation technique than without it on various DMControl tasks.
strength_weakness: strength_weakness: 1. **The story seems improperly situated in the data augmentation literature.** While it is true that most prior data augmentation works focus on perturbation-based augmentations, many prior works have exploited task symmetries and invariances to generate additional data that agrees with the task's dynamics and reward function [1-6]. Corrado et. al [1] calls these dynamics-invariant augmentations.

2. **It’s unclear if this change of representation is necessary for data augmentation.** Corrado et. al [1-2] perform data augmentation using SO(3) rotations in MuJoCo tasks similar to those studied in this paper *without* changing the task’s state features.

3. **Weak empirical results.** In 5/9 tasks, DDPG + data augmentation performs just as well as DDPG without data augmentation. With 5 seeds per curve, I have low confidence in the observed benefits in the remaining tasks; RL is notoriously high variance, and the variance between runs is enough to create statistically different distributions. 95% confidence belts assume the distributions of returns at each evaluation point are normally distributed, which is likely not the case. 

[1] Corrado & Hanna. "Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates." ICLR 2024.

[2] Corrado et. al. "Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning." arXiv:2310.18247

[3] Pitis et. al. “Counterfactual Data Augmentation using Locally Factored Dynamics.” NeurIPS 2020.

[4] Pavlov et. al. "Run, Skeleton, Run: Skeletal Model in a Physics-Based Simulation." AAAI 2018.

[5] Abdolhosseini et. al. “On Learning Symmetric Locomotion.” ACM SIGGRAPH 2019.

[6] Adrychowicz et. al. "Hindsight Experience Replay." NeurIPS 2017.

[7] Henderson et. al. "Deep Reinforcement Learning that Matters." AAAI 2018

**Minor Comments**
1. I think section 3.2 can be omitted; the augmentation framework described in section 4.3 can be applied to any off-policy RL algorithm.
    
6. Line 271: typo, I think the second $s_t$ should be $s_t\prime$
questions: questions: 1. Line 40-43: Could the authors elaborate on what “uncorrelated” means here?

4. Line 159: Does anything change if the morphology tree contains a cycle?
    
5. Line 272: What does it mean to keep $a_t$ and $r_t$ invariant?
limitations: limitations: The paper addresses limitations, though I think an additional limitation should be emphasized: not only may itbee difficult to specify a symmetry, symmetries might not even exist in some tasks. 

I don't believe this limitation takes anything away from the paper though; it's a core limitation of most non-perturbation-based data augmentation methods, and it's fine if these methods only apply to some tasks. That's simply the nature of data augmentation. Thankfully, in many real-world tasks (especially robotics tasks where you have an agent acting in 3D space) a human *can* identify symmetries. Pitis et. al [1] and Corrado et. al [2] discuss this point too.

[1] Pitis et. al. “Counterfactual Data Augmentation using Locally Factored Dynamics.” NeurIPS 2020.

[2] Corrado & Hanna. "Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates." ICLR 2024.
paper_summary: paper_summary: This paper introduces a novel data augmentation strategy tailored for reinforcement learning (RL) agents operating in state-based continuous control environments. The method leverages limb-based state features rather than joint-based configurations, allowing for more effective augmentation. Experiments conducted on various tasks from the DeepMind Control Suite demonstrate significant improvements in data efficiency and performance over standard RL algorithms and existing augmentation methods.
strength_weakness: strength_weakness: Firstly, I am not an expert in this research field. I think this paper lacks some deeper theoretical support. The scenario covered by Theorem 1 is overly simplistic. The proposed method involves additional computations for applying Euclidean transformations and managing the limb-based state features. The paper could benefit from a more detailed discussion on the computational overhead and potential optimizations.
questions: questions: Could the authors provide more details on how to tune the various parameters involved in the proposed method, such as the choice of Euclidean transformations and the proportion of augmented data?
limitations: limitations: The authors acknowledge the need for task-specific tuning of the hyperparameter and the requirement for knowledge of strict Euclidean symmetries. They have adequately addressed the limitations.
paper_summary: paper_summary: This paper introduces a novel data augmentation method for reinforcement learning in continuous control tasks. The key innovation is leveraging Euclidean symmetries inherent in these tasks by applying rotational transformations to the original states. The authors propose using a limb-based state representation instead of the standard joint-based one to make states more amenable to these transformations. Integrated with the DDPG algorithm, this method is evaluated on 10 tasks from the DeepMind Control Suite, demonstrating improvements in data efficiency and performance, especially for complex 3D tasks. The approach outperforms existing data augmentation techniques and proves computationally efficient compared to alternative methods.
strength_weakness: strength_weakness: 1. The paper doesn't clearly address how the method handles environments where Euclidean data augmentation might violate constraints (e.g., obstacles in AntMaze from MuJoCo).
2. The data composition process is not fully explained. Section 4.3 doesn't specify if $B_\text{aug}$ transitions are added to $B$ or substitute points in $B$. 
3. The study is limited to 0-100% augmentation, while exploring larger multipliers could be insightful as one transition can be rotated with different angles
4. The focus is primarily on rotation transformations. Exploring other transformations like translation and reflection could provide a more comprehensive analysis.
5. In 5 out of 9 tasks, $ρ_\text{aug}$ = 0% achieves the best performance, suggesting the limb-based representation may be more critical than the data augmentation itself. However, the choice of limb-based representation seems motivated primarily by enabling Euclidean DA, which may limit generalizability. In addition, the advantages of limb-based over joint-based representations are not thoroughly justified.
questions: questions: 1. There appears to be a typo in line 271 - it should likely read "transform states $s_t$ and $s_{t+1}$".
2. The description of rotations in Section 4.3 is confusing. Why are there two separate rotation steps mentioned?
3. Some recent state-based data augmentation methods are not well cited in the related work, including:
     - Hindsight experience replay (Andrychowicz et al., 2017)
     - Counterfactual data augmentation (Pitis et al., 2020)
     - MoCoDA (Pitis et al., 2022)
     - Guided Data Augmentation (GuDA) (Corrado et al., 2024)
Particularly, GuDA, while focusing on offline-RL and including human guidance, describes a method for state-based data augmentation, including rotation, transformation, and reflection. Discussing these works could strengthen the paper's positioning in the field.
4. The paper focuses solely on the DDPG algorithm. Exploring compatibility with other popular RL algorithms would increase impact.
limitations: limitations: The authors adequately address limitations, noting the need for task-specific tuning and knowledge of strict Euclidean symmetries. They provide constructive suggestions for future work. Regarding societal impact, while the immediate concerns are limited for simulated tasks, a brief discussion on long-term implications for robotics, especially on real robots, could be valuable.

Comments: title: Updates on the suggested experiments (3/3)
Updates on the suggested experiments (2/3)
Updates on the suggested experiments (1/3)
Response to rebuttal
Thank you
comment: Also appreciate your efforts. Best wishes!
Thanks.

As the discussion period comes to its end, we authors thank you for your reviewing!
Thank you for your responses. I still maintain my opinion about the limited effectiveness of the method, given that:

1. The performance boost requires some specific task properties, i.e., rotation symmetries about gravity.

2. Even for tasks where the method demonstrates improvement, it requires:
a) Changes in configurations
b) Additional hyperparameter tuning

3. I still question the difference between the DDPG performance and the performance reported in [1] Figure 5. This might lead to marginal performance differences between the proposed method and the default DDPG setting if we choose hyperparameters carefully.

However, I appreciate the authors' additional experiments and final response, which, based on the paper, do help us better understand the method. Therefore, I have increased my score accordingly.

[1] Tassa, Yuval, et al. "Deepmind control suite." arXiv preprint arXiv:1801.00690 (2018).
Dear reviewer,

Thank you and we appreciate your updated assessment!

We make two additional comments based on your feedback.

1. We would like to emphasize that we did not cherry pick tasks where our method outperforms others; instead, we sampled a range of tasks with various degrees of symmetries to show when and why our method can bring benefits.
Our results show clear advantages of our method on tasks that exhibit rotation symmetries about gravity (e.g., 2D Reacher_hard, most 3D tasks), where doing *both* limb-based representation and a $\rho$ >0% on top of it is crucial to achieve the best performance.
For other 2D tasks, it is expected that $\rho$ >0% will bring limited benefits over $\rho$ =0%, as we have explained. We plan to move these experiments to appendix in our revision, leaving room to incorporate the discussion and new results during rebuttal.

2. It is pretty much expected that the performance will saturate as multipliers $\rho$ or $N$ gets larger and that the optimal multiplier is task-dependent, which is also observed in prior work, including the very recent Corrado & Hanna, ICLR 2024.
This issue holds true equally for our novel data transformation and for other transformations. 
As discussed in Limitations, we do not claim contributions on tuning this multiplier, which should be a separate contribution that would benefit our augmentation method as well as others.
Thank you for providing the new results. If I understand correctly, these results suggest that a large multiplier N does not offer significant benefits compared to the optimal $\rho_\text{aug}$ reported in the paper. My comparison focused on new results and $\rho_\text{aug}$ = 50% for Hopper3D_hop and $\rho_\text{aug}$ = 100% for Humanoid_run.

The purpose of this comparison was to understand the relative contributions of proper representation (i.e., limb-based) versus Euclidean augmentation to performance improvement. Combining the new results with Figures 3 and 4 from the original paper, it appears that:
1. In 7 out of 10 environments (Cheetah_run, Cheetah3D_run, Hopper_hop, Hopper3D_hop, Walker_run, Walker3D_run, Reacher_Hard), DDPG+Ours ($\rho_\text{aug}$ = 0%) achieves near-optimal performance. This suggests that the change in representation alone is beneficial.

2. In 6 out of 10 environments (Cheetah_run, Cheetah3D_run, Hopper_hop, Walker_run, Quadruped_run, Reacher_hard), standard DDPG achieves near-optimal performance. In addition, the DDPG performance reported in this paper appears to be lower than that reported in the DeepMind Control Suite paper [1], Figure 5.

These observations raise concerns about the method's effectiveness: The performance improvement appears marginal and requires careful selection of the $\rho$ value for each task. Increasing the amount of data does not seem to provide additional benefits, which undermines the motivation for using this as a data augmentation method. It is not guaranteed to outperform default DDPG without changes to the representation configuration.

I do appreciate authors' efforts on addressing my questions, but given these concerns, I maintain my original assessment and score.
For Humanoid_run, the results in our submission have shown that a multiplier $\rho_{\rm aug}$=100% is better than $\rho_{\rm aug}$<100%, so we keep using $B$=256 and have finished running $N$=3, 5, with the results shown below (we report mean and std_err over 5 seeds at 5M steps).
| No Augmentation, N=1 | No Augmentation, N=3 | No Augmentation, N=5 | Our Augmentation, N=3 | Our Augmentation, N=5 | Our Augmentation, $\rho_{\rm aug}$=100% |
|----------------------|----------------------|----------------------|-----------------------|-----------------------|------------------------------|
| 89.19 (22.16)        | 99.31 (24.65)        | 70.82 (28.99)        | 103.93 (25.61)        | 109.46 (7.15)         | 158.55 (15.13)               |


We make the following observations:
- Our method with the best rho_aug=100% is still better than the variants with $N$=3, 5.
- The difference between $N$=3 and $N$=5 is not significant

To summarize, we did not observe additional benefits of using multipliers larger than 100%, at least not with the straightforward variants we have experimented with.
This might be due to the fact that, although we can generate large amounts of augmented trajectories with nearly zero cost, these “fake” trajectories are equivalent to the real ones up to a rotation, so their “marginal effects” can drop quickly after a small multiplier.
Due to the time limit, we have run these DDPG variants on the tasks of Hopper3D_hop and Humanoid_run, where our method has brought significant improvements.
For Hopper3D_hop, the results in our submission have shown that a multiplier $\rho_{\rm aug}$<100% is better than $\rho_{\rm aug}$=100%.
We conjecture that a smaller batch size $B$ will favor larger multipliers, so we have run $B$=32 with $N$=1, 2, 3, 5,10 with the results shown below (we report mean and std_err over 5 seeds at 2M steps). 
| N=1, No Augmentation | N=2, No Augmentation           | N=3, No Augmentation           | N=5, No Augmentation           | N=10, No Augmentation          | N=2, Our Augmentation | N=3, Our Augmentation | N=5, Our Augmentation | N=10, Our Augmentation |
|---------------------------|---------------|---------------|---------------|---------------|------------------------|------------------------|------------------------|-------------------------|
| 43.02 (17.43) | 43.14 (4.39) | 55.14 (6.25) | 42.83 (3.03) | 32.25 (8.98) | 253.14 (13.65) | 252.15 (10.06) | 250.71 (6.21) | 212.92 (7.09) |


For reference, the results for $B$=256 (in the submission) is shown below.
| N=1, No Augmentation (Standard DDPG) | $\rho_{\rm aug}$ = 50%, Our Augmentation |
|---------------|------------------------------|
| 27.18 (11.12) | 261.35 (15.27)               |


We make the following observations:
- In terms of performance, the smaller batch size of $B$=32 is qualitatively similar to $B$=256. In both cases, standard DDPG cannot learn effective policies, even with $N$>1 inner iterations, while our augmentation methods can.
- For the new experiments of data augmentation with $B$=32 and $N$>1, performance peaks at $N$=2,3 and drops with larger $N$. This suggests that using a very large $N$ is not helpful here.
Dear reviewer,

We have conducted additional experiments as suggested.
The primary request is to investigate whether it is beneficial to rotate transitions in a batch with different angles, as the submission only has “multipliers” up to 100%.

To fulfill this request, we have run the following DDPG variants based on our limb-based data augmentation method:
- Sample a batch of $B$ limb-based transitions
- Perform $N(>1)$ iterations of loss computations and gradient updates
    - For the first iterations, use the original, un-rotated $B$ transitions
    - For each of the rest $N-1$ iterations, rotate all $B$ transitions with a random angle per transition

For fair comparison, we have run similar DDPG variants of $N$ “inner-loop” iterations with the original, joint-based representation:
- Sample a batch of $B$ transitions of join-based representations
- Perform $N(\geq 1)$ iterations of loss computations and gradient updates using the same B transitions   

This way, the standard DDPG in our submission is the (B=256, N=1) case.
Thank you!
Thanks for the responses to my questions. I am sticking to my score.
Thank you!
We will adopt your suggestion on "uncorrelated".
Thank you for the detailed response! I'm glad to see the comparisons with the prior works; including a some discussion on them would better situate this paper in the literature. It would also emphasizes some of the novelty by clarifying how these augmentations differ from those in prior works, particularly this part of your response:

> However, we hypothesized this data augmentation by rotating original joint-based features would bring little benefit to locomotion tasks. This is because torso features make up only a fraction of all features when the number of limbs is large (which is the case in our tasks). Our limb-based representation instead rotates all limbs to provide richer augmentation.

Regarding experiments: It would be clearer just have the 3D tasks where augmentation helps in the main paper. You can explain that these augmentations would not be helpful in the 2D tasks, and then point the reader to an appendix containing the 2D experiments. I have raised my score because of these clarifications.

All of my questions have been answered. Just a quick follow-up comment: I think a clearer alternative to "uncorrelated" would be to say something like "the augmented data generally does not agree with the task's dynamics" or "the augmented data is generally not dynamics-invariant."
Thank you!
Dear reviewer,

As the discussion deadline approaches, we would like to know if our response has addressed your concerns. Should any concerns remain, we will gladly address them.

Thank you again for reviewing our paper.
Dear reviewer,

As the discussion deadline approaches, we would like to know if our response has addressed your concerns. Should any concerns remain, we will gladly address them.

Thank you again for reviewing our paper.
Thank you for the response. Since I am not an expert in this field, I will maintain my current neutral score. Thank you again or your detailed responses to my review.
Thank you! We are glad our reponse have addressed your concerns.

We are working on the suggested experiments and will update you on our progress by discussion deadline.
We agree that they will provide additional insights.
Thanks for the clear feedback. I think they have addressed most of my concerns. 

If time permits, I would still highly recommend running **"W3: The study is limited to 0-100% augmentation, while exploring larger multipliers could be insightful as one transition can be rotated with different angles."** It makes sense to ensure it is a substitution for a fair comparison with the baseline as mentioned in the response of weakness 2. However, in terms of understanding how far this method is beneficial on top of the base algorithm—given that collecting real trajectories is expensive while data augmentation is nearly zero cost—I think it would be interesting and would definitely make the paper stronger to see that for N real collected trajectories, [1, 10, 50, 100, ...] * N through augmentation on the N trajectories outperforms the base algorithm.