Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning
Authors: Yunpeng Jiang
Jianshu Hu
Paul Weng
Yutong Ban
Venue: NeurIPS 2025 poster ()
Keywords: Deep Reinforcement Learning
Robotics Control
Primary area: reinforcement_learning

Abstract: Symmetry is pervasive in robotics and has been widely exploited to improve sample efficiency in deep reinforcement learning (DRL). However, existing approaches primarily focus on spatial symmetries—such as reflection, rotation, and translation—while largely neglecting temporal symmetries. To address this gap, we explore time reversal symmetry, a form of temporal symmetry commonly found in robotics tasks such as door opening and closing. We propose Time Reversal symmetry enhanced Deep Reinforcement Learning (TR-DRL), a framework that combines trajectory reversal augmentation and time reversal guided reward shaping to efficiently solve temporally symmetric tasks. Our method generates reversed transitions from fully reversible transitions, identified by a proposed dynamics-consistent filter, to augment the training data. For partially reversible transitions, we apply reward shaping to guide learning, according to successful trajectories from the reversed task. Extensive experiments on the Robosuite and MetaWorld benchmarks demonstrate that TR-DRL is effective in both single-task and multi-task settings, achieving higher sample efficiency and stronger final performance compared to baseline methods.

Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning
Yunpeng Jiang jyp9961 11999610sjtur @sjtu. edu. cn Global College Shanghai Jiao Tong University
Jianshu Hu hj hj919988s tu. edu. cn Global College Shanghai Jiao Tong University
Paul Weng"
Yutong Ban" ybanSsjtu. edu. en Global College Shanghai Jiao Tong University
paul weng@dukekunsharl edu. en Digital Innovation Research Center Duke Kunshan University
Abstract
Symmetry is pervasive in robotics and has been widely exploited to improve sample efficiency in deep reinforcement learning (DRL). However, existing approaches as reflection, rotation, and trans-
primarily focus on spatial symmetries- lation-wii -while largely neglecting temporal symmetries. To address this gap, we explore time reversal symmetry, form of temporal symmetry commonly found in robotics tasks such as door opening and closing. We propose Time Reversal symmetry enhanced Deep Reinforcement Learning (TR-DRL)), framework that combines trajectory reversal augmentation and time reversal guided reward shaping to efficiently solve temporally symmetric tasks. Our method generates reversed transitions from fully reversible transitions, identified by a proposed dynamics- consistent filter, to augment the training data. For partially reversible transitions, we apply reward shaping to guide learning, according to successful trajectories from the reversed task. Extensive experiments on the Robosuite and MetaWorld benchmarks demonstrate that TR-DRL is effective in both single-task and multi- task settings, achieving higher sample efficiency and stronger final performance compared to baseline methods. Our project website and source code can be found in 1 andd
Introduction
Deep reinforcement learning (DRL) is a powerful machine learning framework capable of solving complex tasks, with applications across robotics, quantitative trading, and video games. Despite its successes, DRL often suffers from low sample efficiency and poor agent robustness. To address these challenges, symmetry, common property in many real-world scenarios, has been leveraged to improve both sample efficiency and agent performance. Symmetry can be used to augment trajectories collected during training in both state-based (Lin et al., [2020 Kidziñski et al. 2018 and image- based settings Yarrtt al.) 2002]. Alternatively, symmetry can be embedded directly into the network architecture, making it an inherent property of the model [Cohen and Welling 2016, Wang et al.
2022). In addition, can be enforced as regularization term [Hu et al. 2024a Raileanu 2021
Corresponding authors.
I Projecc
Soorce Code:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Dynamics and Potential
Learning
Time Reversal DRL
Drawer Open
Reversal Augmentation
Reverse-guided Reward Shaping
Drawer Close
Figure 1: For a task pair, the proposed TR-DRL framework learns dynamics and potential models, leverages trajectory reversal augmentation with dynamics aware filtering and time reversal symmetry guided reward shaping, and boosts sample efficiency in both tasks.
However, existing work (see Related Work in Section predominantly focuses on spatial symmetries, such translation, reflection, and rotation, while temporal symmetries, including time-reversal symmetry and time dilation, remain largely underexplored. Intuitively, time-reversal symmetry corresponds reflection with respect time, assuming actions be reversed, which often holds in navigation tasks. Time dilation occurs in certain robotics control problems when the agent can control the speed of action execution [Hu et al. 2024b]. In this paper, focus leveraging time- reversal symmetry in robot manipulation tasks, where the agent controls the position and orientation of the end-effector. Unlike spatial symmetries, where augmented samples typically remain valid, temporally reversed transitions may result in invalid transitions due to complex interactions between the robot and objects.
Consider a task pair, door opening outward and door closing from outward. An augmented trajectory of closing door from outward (Figure 2] from right to left) can be generated by reversing trajectory where the agent opens the door by grasping and moving the handle outward (Figure ([a) from left right). In this case, the state pairs within the trajectory are fully reversible. Then consider another task pair, door opening inward and door closing from inward. When the agent closes the door by simply pushing it without grasping the handle (Figure (b)), reversing the trajectory becomes nontrivial. This is because the agent cannot feasibly open the door without first grasping the handle, making the reversed transitions invalid. However, certain components of the state, such as the object state (e.g., the door's opening angle), may still be reversible, even if the full transition is not. Such cases correspond partial reversibility of the transitions where the concept of state decomposition [iitis et al. 2020 enables isolation of dynamically reversible components.
To exploit (partial full) time reversal symmetry in DRL, we propose general framework (see Figure [ that incorporates two complementary techniques, which can accelerate training for a pair of related tasks. For full time reversal symmetry, learn inverse dynamics model obtain the reversed actions and generate the augmented transitions when training in both tasks. To ensure the validity of these reversed transitions, we additionally train forward dynamics model to filter out transitions that violate the true system dynamics. For partial time reversal symmetry, the reversible component of the state can be intuitively used guide policy learning. We leverage this form of symmetry through reward shaping, encouraging the agent for one task to follow trajectories that resemble the reversed versions of successful trajectories from the other task.
Contributions Our contributions can be summarized follows:
(i) Based on (full) time reversal symmetry (Section 3). we introduce the novel notion of partial time reversal symmetry (Section 4.1) to exploit temporal symmetry in more general settings (e.g., when objects are pushed).
(ii) We propose two techniques (Sections 4.2 and to exploit time reversal symmetry:
For full time reversal symmetry, transitions identified as reversible by a trained dynamics-aware
filter are augmented improve the sample efficiency of DRL algorithms.
For partial time reversal symmetry, a reward shaping mechanism exploits transitions from successful trajectories to guide the training of the DRL agent.

---

Figure 2: Examples of fully and partially reversible trajectories. (a) Fully reversible: An example of opening the door outward by grasping the handle; (b) Partially reversible: An example of closing the door from inward by pushing the door.
(iii) We conduct extensive experiments (Section 5 standard robotics benchmarks (Robosuite, Metaworld) demonstrating that our approach significantly improves both sample efficiency and final performance compared to baseline methods. An ablation study further validates design choices and highlights the contributions of each component within our framework.
2
Related Works
We summarize related works from two research directions in DRL, which include symmetry, and reward shaping techniques. For symmetry, we divide it into spatial symmetry and time reversal symmetry.
Spatial Symmetry in DRL Spatial symmetry, including reflection, rotation, and translation, are extensively exploited in DRL. These symmetries enable the generation of synthetic transitions from single environment interaction. effectively improving sample efficiency. For example, prior works [Lin et al. 2020, Corrado and Hanna 2024 Corrado et al. 2024] have shown that applying spatial augmentations such as reflection, rotation, and translation significantly boosts sample efficiency in Statebbaee robotics control tasks. In image-basee based RL, translation symmetry has also been widely adopted to enhance performance (Yarats et al. 2022 Ma et al., 2024 Hu et al. 2024a]. Data augmentation methods can also improve robustness to noise [Sinha et al. 2021 Qiao et al. 2021]
Moreover, spatial symmetry can be embedded directly into the neural network architecture through equivariance [Cohen and Welling 2016)] Wang al., 2022 2023], ensuring that the network respects these symmetries by design. This architectural integration reduces training time and improves generalization across diverse inputs. In contrast to these prior efforts, our work focuses on exploiting time-reversal symmetry, a form of temporal symmetry that remains underexplored in DRL.
Time Reversal Symmetry in DRL Time reversal symmetry has been leveraged for data aug- mentation [Barkley et al. 2023) and for learning dynamics-consisterd latent representations from
images [Ceeng et al. 2023 J. contrast to simply negating actions in reversed transitions [Barkley sophisticated strategy to derive reversed actions, making applicable to a broader range of environments. Morcover, our method focuses on state-based control tasks, where full state information is available, eliminating the need of learning latent representations from visual observations.
al. 2023 Yao et al. 2023) our approach employs
Some existing works focus on exploring reversibly from goal states, utilizing the time symmetry to enhance the agent's exploration towards desired states. Starting from a goal state, the agent explores by imagining reversal steps [Edwards et al.) 2018) or predicting preceding states leading to goals [Goyal et al. 2019 Instead using imagined trajectories, true trajectories starting from goal states are given in TRASS [Nair et al.) 2020). and the agent learns from the reversed trajectories. Unlike these works, we leverage time reversal symmetry not only from goal states but for every transition in the trajectory, enabling broader application of time symmetry across the entire state space.
Other prior works focus on enhaneing the reversibility of the agent, exploring strategies ensure that agents can backtrack or reset their actions to avoid irreversible states. For instance, Grinsztajn et al. [202T) propose distinguish reversible from irreversible actions to improve decision-makinn makiin in DRL. This distinction enables agents to prioritize reversible actions that are safer, as they guarantee the ability to backtrack if needed. Furthermore, alongside the normal policy to prevent agents from entering non-reversible states, ensuring safety in exploration phase and achieving better training efficiency. While their reset policy sets initial state as the ending state of the current policy and the goal state the task's starting point, method
(ettermmorreEEsebaaecclee Eysenbach et al. 2018) propose learning a reset policy
3

---

treats two reversible tasks independently. with initial and goal states defined separately for each task. Additionally, our method is orthogonal theirs and can be integrated enhance the training of their reset policy.
Reward Shaping in DRL Reward shaping a powerful technique for enhancing the efficiency of DRL algorithms [lbrahim et al., 2024). as it guides agents toward desired behaviors. The idea of using shaped rewards to guide learning naturally aligns with our objective of leveraging reversed trajectory in tasks of time reversal symmetry. However, reward shaping has not yet been explored in the context of time reversal symmetry. In this work, we exploit time reversal symmetry by training a potential function guided by reversed trajectories. Potential- involves defining potential function over the state space, which captures the agent's is desired progress toward the goal. Importantly, the optimal policy remains unchanged with potential-bases reward shaping, providing a theoretical foundation for its application in our method.
reward shaping [Ng et al. 1999]
3 Background
In this section, we recall the framework of deep reinforcement learning (DRL), the soft actor-critic algorithm, the concept of time reversal symmetry, and potential-basee reward shaping in reinforcement learning (RL).
Deep Reinforcement Learning (DRL) For any set X, ( denotes the set of probability dis- tributions over X. A Markov Decision Process (MDP) model M (S, A. R, T, Po: 7) is defined by a set of state S, a set of action A, a reward function R S x A + R, a transition function T: S x A ASS), a probability distribution over initial states Po E A(S), and discount factor E 1]. In RL, the agent learns a policy x(- s) E ((A) by interacting with the environment, aiming to maximize the expected return J = E,, E 7'ra Su po). where E,, denotes the expectation over л and r is the reward that the agent obtains at each timestep t.
Soft Critic (SAC) Maximum entropy reinforcement learning (RL) addresses standard RL problems using an alternative objective that explicitly encourages stochastic policies. The objective combines cumulative reward with an entropy term: J = Ex 1-t 7'r ft + a/l ( |s]]]. where is the discount factor, a is a trainable coefficient of the entropy term, and H((( st)) represents the entropy of of the policy distribution T(- st). The Soft Actor-Critic (SAC) algorithm (Haarnoja et al.
2018 optimizes this objective by training the actor To and critic Qu with the following losses:
(0) = E,,ND,arz log (a s1) - (st
LQ( B [Q (#,, as) (8t, )}
where Q(+,, ae) = r. + 1 1x+1) - a log Ne (ur+1 18++1). which is the target Q-value com- puted using a target network, and a++1 ~ T: 8 Here, 0, 3 and & represent the parameters of the actor, the critic and the target critic respectively, while D represents the replay buffer. To stabilize training, the weights of the target network are updated as an exponential moving average of the online critic network's 's weights.
Time Reversal Symmetry in DRL Given an involution satisfies (full) time reversal symmetry (adapted from Barkley et al. 2023 if for all 8t, 8t-1 ES,
2 f S x A S S Ax S, an MDP
T(s+41 8t, az) = T(x Ke+1, āe).
(2)
where 13 āe, Be) [st,, 0t2 11) and = denotes time reversal operation on S or A. Intuitively, involution J' represents the symmetry that reverses the passage of time. Note that in some situations, be simply written as ['s a, s') (Js(s), (a))), fs(s')) using an involution fs over states and an involution fA over actions. An example of time reversal symmetry in physical system is the transformation of position p. momentum g. and the applied force a. The involution fs transforms
state s (q. p) into fs(s) = (q. -p), preserving position while negating momentum, which is a common phenomenon in physical systems. For the action a. fA reverses the applied force such that fa( This ensures that the dynamics remain consistent under time reversal.
Recall an involution is one-to-one mapping, which is its inverse.

---

(a) Dynamics and Potential Training
(c) Time Reversal Guided Reward Shaping
Tisk Drawer Open
Sample
Symmeetry gu Potentall
TR-DRL Policy
Trijctinie
dynomics Model
Sp+1
Pcet
eynamics Model
Time Reversal Symmettric Task Pairs
Merge
St
Ste
Symmeltry- guidedd Potential
Potential
Reversed Actions
Forward Model
Trajectoiie
Dynamice -award Filtering
Xi
Time
Reverse
dynamics Model
Trajectory Reversal Augmentation
Tisk Drawer Close
(5
Sample
Figure 3: Overview of our TR-DRL. We learn dynamics and potential models, apply reversal augmentation on transitions from the reversed task, and apply time reversal symmetry guided reward shaping on all transitions.
We recall the concept of potential-based reward shaping pro- posed by Ng et [999]. shaping reward function JF S x exists real-valued function & S + R such that for all E S, a E A. s's E,
Fotential-Based Reward Shaping
+ potential-base there
F(s,a, 8') ( (8),
This condition is necessary and sufficient to ensure that an optimal policy of the modified MDP M' = (S, A, R+ F,T, po: remains optimal in the original MDP (S, A, R, T,,o, Po.
Methodology
In this section, first define the problem set-up considered in this paper and introduce two types of time-reversal symmetry, full or partial time reversal symmetries, for which we provide illustrative examples in robotics (Section 4.1p. We then propose generie method, shown in Figure B which applies trajectory reversal augmentation (Section 4.2) on fully reversible transitions identified by our proposed (ynamies-consistent filter, and employs reward shaping (Section 4.3) guided by partially reversible transitions.
4.1 Problem Formulation
In this paper, assume that the RL agent aims at learning to solve (at least) two related tasks in the same environment (e.g., door opening/closing or peg insertion/removal) For such a pair tasks, the RL agent may learn in more data-efficient way by exploiting full time reversal (FTR) symmetry and partial time reversal (PTR) symmetry (see definition below). While Barkley et al.) [2023) assume that the FTR symmetry holds globally and that the involution to transform actions is known, which makes this temporal symmetry property too restrictive and difficult to apply in practice, we do not make these assumptions, which allows us consider scenarios like the next example. However, the challenge now is to detect when Equation 2 holds and learn to recover reverse actions ă
Consider pair of manipulation tasks: peg insertion and peg removal. Assume end-effector position control and that the state includes the positions of both the end-effector and the object (i.e., peg). When the robot arm holds and moves the peg towards the hole, for transition (st, (t, 8t+1), reversing the action enables the agent to move from 8:41 back to 8, without violating the true dynamics. This means that all transitions along this trajectory exhibit FTR symmetry. However, contacts and frictions prevent the definition of involution / and the peg can be dropped, the transitions naturally not reversible anymore.
Example 1.
5

---

While the relaxation of these two assumptions extend the applicability of FTR symmetry, for many pairs of tasks, an even weaker notion of temporal symmetry may be needed. We therefore introduce the novel notion of partial time reversal (PTR) symmetry:
Partial Time Reversal (PTR) Symmetry Assume that a state s E S (resp. s' S) can be decomposed into two parts (z,y) E d' x y (resp. (x',y') E X' x }) and that an involution fx X X is given. A pair of states (s,s') S satisfies PTR symmetry if there exist (T, )22 and (a, E A2 such that:
T(s (, = T(3 ",ā),,
where = fx(x), f (E, and 3 D'). Intuitively, X' is the part that is reversible (e.g., containing object state information). Using this weaker property, we can now account for scenarios like the following example:
Example 2. Consider another pair of tasks: door opening and door closing inward, with a similar definition of state and action spaces in Example In the door closing task, the agent learns to close the door by pushing it, without grasping the handle. Along this trajectory, the transitions do not satisfy FTR symmetry, as there does not exist an action that allows the robot arm to pull the door without grasping the handle. However, the object (i.e. door) state remains reversible. We can find corresponding state pairs with reversed object state in the trajectories of door opening tasks. These pairs reflect PTR symmetry, as only the object component of the state is reversible.
In the next two subsections, we explain how to exploit FTR and PTR symmetries in DRL.
4.2 Trajectory Reversal Augmentation with Dynamics-Aware Filtering
In this section, we introduce how we augment the fully reversible transitions and how these fully reversible transitions are detected by a Syynamics-consistee
filter. Given a pair of tasks with time reversal symmetry, any transition (s, a. s') exhibiting FTR symmetry defined above can be augmented by generating its reversed transition (s', ã, 8) and incorporating it into DRL training. Now the problem to be solved is finding ä. In some robotics tasks, straightforward choice of ä is to negate the action which corresponds to reversing forces or torques, i.e. = -a [Barkley et al. 2023]. However, does not work for tasks involving contact dynamics non-linear effects. To address this, we propose a more general approach by learning an inverse dynamics model h, represented by a neural network:
a = h(s, s'),
(5)
which is trained using transitions collected during RL training by minimizing the following loss:
L = İ (=s.,,s)))D((((ss.. s') a)
(6)
where É is an empirical mean estimating the expectation over the true data distribution and D denotes the replay buffer containing transitions (s, a, s'). Since the pair of reversible tasks share the same underlying dynamics, a single inverse dynamics model can be trained jointly using transitions from both tasks. This shared model ensures the accurate inverse predictions when applied to the reversed task.
Note that trajectory reversal augmentation can only be applied directly on fully reversible transitions. To identify such transitions, additional dynamics-consisten filter is introduced to select appropriate samples from the replay buffer. This filtering is achieved by training a forward dynamics model on transitions from both tasks by minimizing the following loss:
Lg (,,.,,)) a) - s')
This model allows us to verify whether a reversed transition (s', ā, s) is consistent with the underlying dynamics. In particular, for reversed transition (s', ",, s), we feed the state s' and action ă into the forward dynamics model g to get the predicted state
a = _ - g(s h)s,,
(8)
The error between the predicted state å and the true state serves as measure of feasibility. Only when this prediction error & || is below predefined threshold 3, the reversed transition considered valid and included in the training for the reversed task.

---

4.3 Time Reversal Symmetry Guided Reward Shaping
In scenarios where not all transitions fully reversible, trajectory reversal augmentation may become less effective. As an illustration, consider the task mentioned in Example 2 In such cases, most reversed transitions are filtered by the dynamics-consistent filter, since the agent cannot reverse the action (i.e., from "push the door" to "pull the door") without first grasping the handle. However, we can still exploit partial time reversal symmetry to improve sample efficiency. In many tasks, the object state, such as the position of a door or the placement of a peg, remains reversible, while irreversibility arises primarily from the agent state, such joint angles or gripper force.
To exploit this separation, we examine the relationship between object states in the trajectories of pair of partially reversible tasks. Let zy denote the object-related component of the full state St at time step t. Given a high-reward trajectory т = $1. sn) from one task, another trajectory (5o, , 8,) from the reversed task should likewise receive high reward if it achieves the reversed sequence (In, (In.Tn-11: In.1: To); or a partial reward if it accomplishes only a portion of the reversed sequence. This observation raises key question: we leverage the partially reversible time symmetry, such that the reversible object-related components can be used to accelerate agent training?
As an answer to this question, we propose time reversal symmetry guided reward shaping. Here we employ potential-be based reward shaping NNg al. 1999] since it preserves policy optimality and directly operates on states. To fully utilize multiple successful trajectories, we propose to train a potential model & for the reversed task, which maps the object state Ir to potential value (xx)). As discussed, for the reversed trajectory containing sequences from Iy to 2o, the potential values of these object states should increase in the reversed task. Therefore, the object states along this reversed trajectory are labeled with potential values ranging from 0 to 1, and used to train the reversed task. Here, a linear function can be used interpolate potential values between the start and end states. The potential model 4, is then trained to minimize the following loss:
12 - t (b((z))
L = Er (...............
(9)
8t = [01, E (O.....
where B denotes the dataset which includes high-reward trajectories.
Based on Equation 3), the reward for each transition (5g, og 3441, Fr) in the reversed task is reshaped reward shaping mechanism encourages the agent to align the trajectory of object states with the reversed successful trajectory by dynamically
as (x+++) 10te+11)p(200) (2,) during training. This potential-
shaping rewards based on the potential values. the door tasks, for example, a closed-to-oper to-enen trajectory, reversed from successful open-to-closed trajectory, guides the door opening agent by training to predict low potential values for closed door states and high potential values for open door states. Since potential-base reward shaping assigns distinct potential value to each object state within the trajectory, reflecting its proximity to the success state, this smooth progression of potential values improves agent training by offering step-by-step guidance towards the goal state.
4.4 TR-DRL Algorithm
Our proposed techniques to exploit time reversal symmetry can be integrated in various DRL algorithms. For conereteness, an example with SAC is presented in Algorithm [T The training process alternates between two reversible tasks. First, the agents collect data from their respective environments. Then transitions from both environments are used to train the forward and inverse dynamics models. Meanwhile, successful trajectories are employed to update the potential models. During agent training, we augment the original samples from the agent's current task with reversed samples from the reversible task via trajectory reversal augmentation and dynamics-aware filtering. Further, we apply time reversal symmetry guided reward shaping to reshape rewards of all the transitions. Finally, we update the agent with DRL loss.
7

---

Algorithm TR-DRL
Required: pair of reversible tasks (A, B), total number of training episodes N, total number of timesteps in one episode T.
1: Initialize empty replay buffers A and Dg. Initialize actor and critic Q.
2: Initialize potential models. Initialize forward and inverse dynamics models.
3: for N. do
4:
for = 0... do
5: Alternate the following training steps between A and B.
6: Task A:
7:
The agent interacts with the environment and save the transition in replay buffer DA- Update forward and inverse dynamics models using Equation (6) and Equation (7)
8:
9:
Update potential models using Equation (9).
10:
Sample two minibatches d and d from DA and DB--
11:
Generate aug from dæ by reversal augmentation with dynamics-aware filtering. Apply time reversal symmetry guided reward shaping on da J d.gg
12:
Update the actor and critic, and Q, with a U duu using Equation I/ Task B:
14:
end for
16:
end for
Experimental Results
To demonstrate the effectiveness of our proposed method, we conduct comprehensive experiments to assess the performance of our approach both single-task and multi-task settings. We also run ablation study on method to illustrate the design choice of different components.
To validate our method, we evaluate our method in 60 environments from two standard robotics control benchmarks, Meta-World (Yu et al. [020] and Robosuite [Zhu et al.)
Experimental setup
2025
Detail introductions and example figures of these environments are provided in Section A| We
SAC HHaanooj al., 2018). multi-task SAC (Yu et al. 2020], and multi-headed SAC Yu et al.
2020 as the baselines for comparison. Hyperparameters, such as network architecture and learning rates, listed in Section B We sparse rewards in all our experiments, which makes learning challenging for the agent. To mitigate this, we initialize the agent's replay buffer with 10 expert demonstration trajectories, providing guidance agent's exploration. Unless specified, all reported scores are averaged over five runs, with standard deviations included in the results. Throughout each run, the agent is evaluated every 20 training episodes by calculating the average success rate of 20 evaluation episodes. To present the aggregated performance, we compute the inter-quartile mean (IQM) as proposed by Agarwal al. [202T]
5.1 Main results
Robosuite-Single task To demonstrate the efficiency of our proposed method, first evaluate it under single-task setting in Robosuite, where we train an agent for each task. Here, five pairs of tasks exhibiting time reversal symmetry are considered: door opening/closing inward, door opening/closing outward, peg insertion/removal, nut assembly/disassembby and block stacking/unstacking The IQM of agent performance across 10 environments are shown Figure 4 and full evaluation curves are provided in Figure (uuure[4]du [4 due to page limit. The results demonstrate the performance gain of our method over the baseline and confirm the contribution of each component in our method.
Robosuite-Multi task We further evaluate our method in multi-task settings. Our method is orthogonal to existing multi-task learning frameworks, meaning it can be seamlessly integrated with them. To highlight the performance gains of our approach, we demonstrate its effectiveness by combining it with existing multi-task methods. Here, we start with training one agent for a pair of tasks. Later on, we extend our method using a single agent for all concerned tasks. To train single agent for multiple tasks, we consider extend the models to either taking an additional task embedding as input (task-conditioned) or outputting the actions of several tasks at the same time healt,-head).

---

Number of Environment Transitions
X3H Suromm
Method
50k
100k
150k
200k
Single-Taske Task SAC
0.11
0.6240.16
0.93+0.07
0.97++0.03
+reversal aug
0.22++0.10
0.88+0.10
0.99:+0011
1.00+00000
+reversal reward shaping
0.26=+0.10
0.79+t0.12
0.97++0.03
1.00+0.00
044
Single-Task TR-SAC (Ours)
0.33
0.92.10.07
1.00-10.00
1.00
Doo Open
Inaad
Stumenbll
Jeyt let SCC
grgenias.os reratt
vevemna renatt argl-an 1ol4
2814
Sereeto T952
Figure 4: Results for single-task setting in 10 environments from Robosuite. Top: Plots and table for IQM of success rate. Bottom: Curves of success rate in two pair of reversible tasks. 'Single-Task SAC": baseline; "+reversal aug": trajectory reversal augmentation with dynamics-aware filtering; "+reversal reward shaping": time reversal symmetry guided reward shaping.
For task-conditioned setting with only two tasks, we use one-hot encoding for the task embedding. The actor, critic and potential models take both the state and the task embedding as input. Considering that the environment dynamics are identical within each task pair, the pair of tasks share the forward and inverse dynamics models. For multi-header setting with only two tasks, the models output values for both two tasks simultaneously. The performance of integrating our proposed method into these baselines in 10 environments of Robosuite is shown in Figure The full evaluation curves of agent performance are included Figure 25] Our proposed techniques clearly enhance the sample efficiency and improve the final performance when combined with the three baselines.
Number of Environment Transitions
Method
50k
100k
150kk
200k
JOH
Single-Task SAC
0.11:00.07
0.6210011
0.9310007
0.97:10.03
Task-Cond SAC
0.31
0.7110.11
0
0.96:10.04
Multi-Head SAC
0.23+0009
0.75+00.1
0.94
0.97+00033
Single-Task
0.3310.114
0.9210.07
1.0010.00
1.00
TR-SAC (Ours)
orge-on
Teb-Coed
and
Task-Cond TR-SAC (Ours)
0.38+0.177
0.93+0.07
0.99+0.01
1.00=0.00
Multi-Heaa TR-SAC (Ours)
0.54
0.9410.055
0.9910.01
1.00
Figure 5: IQM of success rate for multi-task settings in 10 environments from Robosuite. "Task-Cond" and "Multi-Head" are short for "task-conditioned" and "multi-headed" respectively.
Metaworld-Multti task Furthermore, we evaluate our methods on MT50, a benchmark with 50 environments from Meta-World. Within the 50 tasks, we identify 12 pairs of reversible taks and apply our techniques to these pairs. Here, considering the exploding output dimensions when using multi-head setting for 50 tasks, we remove this baseline. Instead, we introduce another baseline called
SAC. Here, the task embeddings are obtained by applying a pretrained language encoder, called CLIP RRaffrrd et al. 2021], on the language instructions of these tasks. The IQM results for these 12 task pairs are shown in the right of Figure and additional results including the average number of training episodes required to achieve a 100% success rate are included in Table 2 and Figure 26 We also present results for all 50 environments of MT50 in Figure 27 and

---

With our proposed techniques, the agent learns faster and performs better compared to the baselines both reversible tasks and all tasks of MT50.
Figure 28
Number of Environment Transitions
Method
IQM Evall Success
50k
100k
150k
Single-Task SAC
0.33:10.05
0.55
0.66:10.05
Task-Cond SAC
0.44:10.05
0.65=00.05
0.75=10.04
100 9a0000
Lang-Cond SAC
0.63:10.05
0.72=
0.78+0..0
Single-Task
0.38++0.05
0.65++0.05
0.76+00.44
TR-SAC (Ours)
single
0.2
Task-Cond
Cond TV
TR-SAC (Ours)
0.52:10.05
0.73=0.04
0.81
Aati-dand SAC
Lang-Coud
0.66.10.05
0.81 0.82.10.04
Fpacdde DOW0
TR-SAC (Ours)
Figure 6: IQM of success rate for multi-task settings in 12 pair of reversible tasks in MT50 of Meta-World. "Task-Cond and LLng-Cood are short for "lask-conditioned" and "language- conditioned" respectively.
5.2 Ablation study
Trajectory reversal augmentation with dynamics-aware filtering We analyze trajectory reversal augmentation with dynamics-aware filtering on three pairs of tasks: door opening/closing inward, door opening/closing outward, and peg insertion/removal. As shown in Section C incorporating reversed transitions improves the performance for fully reversible tasks (e.g., door opening/closing outward and peg insertion/removal) However, most transitions in door opening/closing inward are not fully reversible. Including the reversed transitions generated by the inverse dynamics model leads infeasible transitions, resulting degraded performance. After incorporating dynamics- aware filtering, which removes invalid reversed transitions, the performance surpasses the baseline for partially reversible tasks, demonstrating the effectiveness our filtering strategy. As for the hyperparameter that controls the filtering error tolerance, we finalize its value as 0.01 after tuning among [0.01, 0.001, 0.0001], with the related results presented in Section
Time reversal symmetry guided reward shaping Here we investigate the design choice for the time reversal symmetry-guiddd reward shaping. We first explore how the potential models should be trained. As shown in Section F it is concluded that two potential models should be trained with successful trajectories from the task itself, and from its reversible counterpart respectively. Under this setting, the average of the rewards from these two models used the final reward. Moreover, four different types of potential value functions along the successful trajectories are compared. The linear function outperforms the other choices, shown in Section G
6 Conclusion
We propose TR-DRL, a framework leveraging time reversal symmetry to enhance sample efficiency of DRL algorithms. Key contributions include novel notion of partial time reversal symmetry. trajectory reversal augmentation with dynamics-awan filtering, and symmetry- guided reward shaping. Experiments on Robosuite and Metaworld demonstrate improved agent performance and learning effi- ciency. Future work may explore using prediction errors to identify reversible task pairs automatically. which allows deep reinforcement learning roboties to more efficient.
10

---

Acknowledgments
This work has been supported by the program of National Natural Science Foundation of China (No. 62176154), by the program of National Natural Science Foundation of China (No. 6250020129), and by Shanghai Magnolia Funding Pujiang Program (No. 23PJ1404400).
References
Yijiong Lin, Jiancong Huang, Matthieu Zimmer, Yisheng Guan, Juan Rojas, and Paul Weng. Invariant transform experience replay: Data augmentation for deep reinforcement learning. IEEE Robotics and Automation Letters, 5(4):6615 October 2020. ISSN 2377-3774. doi: 10.1109/1ra.020.. 1109/LRA. 30139977
3013937. URL http:/ /xx doi 1110///.............
Eukasz Kidziński. Sharada Prasanna Mohanty, Carmichael F. Ong. Zhewei Huang, Shuchang Zhou, Anton Pechenko, Adam Stelmaszczyk, Piotr Jarosik, Mikhail Pavlov, Sergey Kolesnikov, Sergey Plis, Zhibo Chen, Zhizheng Zhang. Jiale Chen, Jun Shi, Zhuobin Zheng, Chun Yuan, Zhihui Lin, Henryk Michalewski, Piotr Milos, Blazej Osinski, Andrew Melnik, Malte Schilling, Helge Ritter, Sean F. Carroll, Jennifer Hicks, Sergey Levine, Marcel Salathé, and Scott Delp. Learning to run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments. In Sergio Escalera and Markus Weimer, editors, The NIPS '17 Competition: Building Intelligent Systems, pages 121-153, Cham, 2018. Springer International Publishing. ISBN 978-3-319-94042-7
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved augmentd reinforcement learning. In International Conference on Learning Representations, 2022. /tLL/tttgs//ppppnrrssseeeee//eap
Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, page 2990-2999, JMgR.org, 2016.
Dian Wang, Robin Walters, and Robert Platt. In International Conference on Learning Representations, 2022. URL
reinforcement learning.
net mett/forumiid=7F9cchaav
Jianshu Hu, Yunpeng Jiang, and Paul Weng. Revisiting data augmentation in deep reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024a. URL
https://openreeiex:
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Au- tomatic data augmentation for generalization in reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5402-5415. Curran Associates, Inc., 2021. URL httph: pttttps:///proceeddng iewp:/oceeeiiingeeeeii.pepp.pepp nwwrins......m. wttpppp..cpppppppp wwwpcpprpppppppppeepte feaper_fillee www.ffillieeeeeee Filles/paper//22///ilee 2b3322ffa4807777666ec666------23
Jianshu Hu, Paul Weng, and Yutong Ban. State-novelty guided action persistence in deep reinforce-
ment learning, 2024b. URL htt.httss///aiiv.oottt www.axiw..og/bb.... 200gg999.//409......
Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally
factored dynamics, 2020. URL httpps://axxi..org///aa///
02863
Nicholas Corrado and Josiah P. Hanna. Understanding when dynamics-invariant data augmentations benefit model-free reinforcement learning updates. In The Twelfth International Conference on Learning Representationss 2024. URL ptttpp::/oppereeeee htwppe/nwweee..eeefoouurlolllle. idw.orum/m/:aa222555777777
Nicholas E. Corrado, Yuxiao Qu, John U. Balis, Adam Labiosa, and Josiah P. Hanna. Guided data augmentation for offline reinforcement learning and imitation learning. In Reinforcement Learning Conference, 2024. /tL.attts///opeereiiie
Guozheng Ma, Zhen Wang. Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual reinforcement learning, 2024. URLhtttpt 7/arxiv. org/ abs abs//210.. 04561

---

Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4r1: Surprisingly simple self-supervision for offline reinforcement learning, 2021. URL / org/ abs/
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Efficient differentiable simulation of articulated bodies, 2021. URL BtRL:ttts:://att
org abs
Dian Wang, Jung Yeon Park, Neel Sortur, Lawson L.S. Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry. In The Eleventh International Conference on Learning Representations, 2023. URL https:: netn/ (forun?id-P4MUGRMAL
Brett Barkley, Amy Zhang, and David Fridovich-Keil. An investigation of time reversal symmetry in reinforcement learning. In Conference on Learning for Dynamics & Control, 2023. URL
www.nti httpss//app..mmmatttccbbb...
Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Zhang, Youfang Lin, Shou cheng Song, Han Wang, and Li Jiang. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline RL. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Xiangtong Yao, Zhenshan Bing, Genghang Zhuang, Kejia Chen, Hongkuan Zhou, Kai Huang, and Alois Knoll. Learning from symmetry: Meta-reinforcemment learning with symmetrical behaviors and language instructions, 2023. UtL.httts://azziv.ooo www.oo/gg/////////ee//oooeeeee// abs/ attp//bb///2222/// 10656
Ashley D. Edwards, Laura Downs, and James C. Davidson. Forward-backward reinforcement learning, 2018. Wttp:htttss///xxii org/ abaaabsssaa 180../110.... 10227
Anirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, and Yoshua Bengio. Recall traces: Backtracking models for efficient reinforcement learning. In International Conference on Learning Representations, 2019. URL http::://openreveee.eeee//ooo///
Suraj Nair, Mohammad Babacizadeh, Chelsea Finn, Sergey Levine, and Vikash Kumar. Trass: Time reversal as self-supervision In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 115-121, 2020. doi: 10. /ttp://CCKAAA99222200169996
Nathan Grinsztajn, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. There is no turning back: self-supervised approach for reversibility-awar reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Infor- mation Processing Systems, 2021. URL http:tt//ppptsreeee
www.:/eeefoooooo fost/fooumPPe ? 3X65ea54PP5
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations, 2018. URL https:://ppeerreieett http://ppaneeerrreeeeeerroaoaaaa
Sinan Ibrahim, Mostafa Mostafa, Ali Jnadi, Hadi Salloum, and Pavel Osinenko. Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications. IEFE Access, 011775577-155500 2024. doi: 10.1
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Pruceedings of the Sixteenth International Confer- ence on Machine Leurning, ICML '99, page 278-287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018. URL htttps://proceeddnngg mlr wwwpsrrsssseeeee v80 haarnojaan.. html
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Mota-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
12

---

In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Con- ference on Robot Learning. volume 100 of Proceedings of Machine Learning Research, pages 1094-1100. PMIR, 30 Oct-01 Nov 2020. URL httpps://proceedinges mlr. wtw.p.........0 yu20a.html
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martín, Abhishek Joshi, Kevin Lin, Abhiram Maddukuri, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning, 2025. URL httts://aaxiiv /arxiv www.xxi..rggabe///ee org/abs/2009
12293
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 18-24 Jul 2021. URL aattps://pres htttp::/proceeddngss mlr. press/ httpss/vvv9/raaaea raddlfddfaddaaaa. html.
Ahmed Hendawy, Jan Peters, and Carlo D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In Twelfth International Conference on Learning Representations (ICLR), 2024. URL /arxiv.. org/ abs/231/1111 11385
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce our first and second contributions in Section and the third contribution in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section M
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include the pseudocode of our proposed method and all hyperparameters required to reproduce experimental results in Algorithm I and Section B
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[Yes]
Justification: the camera-ready version, we include the link to our source code repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ Sow.ldesss/oSSSeemiiseebe
www.SSubmisiooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We introduce the environments that we use in Section A and provide all hyperparameters Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide inter-quartile mean (IQM) for aggregated performance with 95% confidence interval.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have stated our compute resources in Section L
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read the NeurlPS Code of Ethics carefully and conformed with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have stated the broader impacts of our paper in Section N
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In our work, we do experiments on robosuite and Meta- World and have cited them properly in the paragraph of experimental setup in Section 5
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research. The core method development in our work does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy pettps://neuriin http://urripa.cccoooeeeeeeeeeeee wctcooofereencee//2255////////// for what should or should not be described.
20

---

Figure 7: Door closing inward.
Figure 8: Door open- ing inward.
Figure 9: Door closing outward.
Figure 10: Door open- ing outward.
Figure 11: Peg inser-
Figure 12:
Peg
re-
Figure 13: Nut assem-
Figure
14: Nut disas-
tion.
moval.
bly.
sembly.
A Environments
We introduce the environments utilized in our Robosuite experiments.
Door Opening/Closing Inward: The agent needs to open/close the door inward. "Inward" means that the door is on the same side the robotics arm. The agent can close the door by pushing it. To open the door, the agent has to grasp the handle and pull the handle to a desired position, making this task pair partially time reversal symmetric. Examples are shown Figure 7 Figure
Door Opening/Clooing Outward: The agent needs to open/close the door outward. "Out- ward" indicates that the door is the opposite side as the robotics arm. In this task pair, the agent has to grasp the handle and then open/close the door, making this task pair fully time reversal symmetric. Examples are shown in Figure 9 and Figure 10
Peg Insertion/Removar The agent needs to insert/remove the peg into/out of the hole. Examples are shown Figure and Figure 12
Nut Assembly/Disassembly: The agent needs assemble/dissasemblee the nut. Examples are shown in Figure 13 and Figure [4]
Block Stack/Unstack. The agent needs to either stack small block onto a larger one or unstack by removing the small block.
The introductions of environments that have used in Meta- World experiments can be found in Yu et al.] 2020), which also provides the language instruction for each task.
B Implementation Details
As shown in Table T we present the value of hyperparameters used our experiments. For experiments in Robosuite, we adopt the hyperparameters specified by Haarnoja et al. (2018) In
21

---

the case of experiments on MT50 of Metaworld, we primarily follow the hyperparameter settings
provided by Yu et al. 2020). Furthermore, in our source code 2 include the implementation
of our method built upon MOORE [Hendawy et al. 2024 more recent and advanced baseline compared to SAC for multi-task RI..
Hyperparameter
Robosuite
MetaWorld
hidden depth hidden dimension horizon environment steps replay buffer capacity
2
512
400
500
200
250,000
100,000
250,000
100,000
random steps batch size discount
5,000
2,000
512
128
0.99
0.99
learning rate learning rate (a of SAC)
le-3
3e-4
le-3
3e-4
target network update frequency
2
target network soft-update rate
0.01
0.005
actor update frequency actor log stddev bounds init temperature
2
[-10, 2]
[-20, 2]
0.1
0.1
Table Hyperparameters used in our experiments.
C Ablation Study of Dynamics-Aware Filtering in Trajectory Reversal Augmentation
As shown in Figure incorporating reversed transitions improves the performance for fully re- versible tasks (e.g., door opening/closing outward and peg insertion/remova)). However, most transitions in door opening/closing inward are not fully reversible. Including the reversed transitions generated by the inverse dynamics model leads to infeasible transitions, resulting in degraded perfor- mance. After incorporating dynamics-awar filtering, which removes invalid reversed transitions, the performance surpasses the baseline for partially reversible tasks, demonstrating the effectiveness of our filtering strategy.
D Hyperparameter Tuning in Dynamics-Aware Filtering
Here, perform hyperparameter tuning of 3 in dynamics-aware filtering. Recall that 3 governs the error tolerance for reversed transitions: = 0.01 filters out transitions where || s & || 0.01 ||ssaxx smiull, where $max and Smin represent the state space extremes. We conduct linear search for 3 among [0.01, 0.001, 0.0001]. Evaluation curves of agent success rate in these six environments shown in Figure 16] while the inter- -utariuuatile mean of agent success rate is presented in Figure 17 Based on these results, we select 3 = 0.01 for subsequent experiments.
E
Separate Plots of Figure 5 and Figure
Due to the page limit in the main text, we combine the results of all methods into single plot for Figure 5 and Figure 6 As shown in Figure [8] and Figure 19 we include separate plots for each algorithm pair (TR vs. no TR) for better readability.
Ablation Study of Potential Models
In our ablation study of potential models, evaluated four training strategies: (1) using only the task's own successful trajectories to train one potential model, (2) using only the reversible task's trajectories to train one potential model, (3) training joint potential model with successful trajectories
22

---

Dasr
Peg
4.3
0.0
3.5
kinglr SAC
1iig bit LSC
230
201
120
230
225
300
200
Door C tue Inear0
Doot Close Outword
P0g Rerroval
Veene aup vio Filer
Figure 15: Evaluation curves of trajectory reversal augmentation with dynamics-aware filtering in 6 environments of robosuite. "Single-Task SAC" serves the baseline. "+reversal aug w/o filter" introduces trajectory reversal augmentation without filtering, while "+reversal aug" incorporates dynamics-aware filtering for reversed transitions.
0.0
6.6
revecallas w/r Si
reveral auy the
imveccal aus will the
rececaala aud beta
reveral say betetled
reveral sat betatled
sclooda
sollecks
sslaecdi
-ostrd aug
noverad aug botum2
:Crerad eap 20
666
04-
04-
reseral
reveral sus bera ubo
Peuetss sid bers
recerall
Testical aud
Figure 16: Evaluation curves of agent success rate using trajectory reversal augmentation with dynamics-awar aware filtering (different (3).
23

---

KIK Eval Success
68
02
Lnda-1e222 mug beta-lee)
ieadriod ong betomion
15 Narlber Eitiiodess letol
Figure 17: IQM of agent success rate using trajectory reversal augmentation with dynamics-aware filtering with different 33
IOM Evol Success
10M 2vail Success
IOM Eval Success
100
Trak Ccdd BRGl
Piul -tead TI-AAC
of ratedderau
20
Figure 18: IQM of success for multi-task settings in 10 environments from Robosuite with separate plots for each algorithm pair (TR vs. no TR). "Task-Cond" and MMuli-HHad are short for "task-conditioned"" and muldi-headded" respectively.
0.-
Single Sea SAC
Balk cand uto.
122
80
Nam Eotoc. FOLE
01000
Figure 19: IQM of success rate for multi-task settings in 12 pair of reversible tasks in MT50 of Meta- World with separate plots for each algorithm pair (TR vs. no TR). "Task-Cond" and "Lang-Cond" are short for "ask-condiitonedd and "language-conddtioneed" respectively.
24

---

Door Open Inward
0.8
Single-Task SAC
+only own
0.2
reversal
+mix own and reversel
shaping
100
200
500
opisodes
Close Inward
1.0
0.8
0.4
Single-Task SAC
+only
0.2
+only reversal
+ mix own and reversal
0.0
+reversal shaping
100
200
300
400
500
episodes
Figure 20: Ablation study potential models the task pair of door opening/closing inward. "Single-Task SAC" serves as the baseline. "+only own" indicates using only the task's 's own successful trajectories to train one potential model. "only reversal" indicates using only the reversible task's trajectories to train one potential model. "mix own and reversal" indicates training a joint potential model with successful trajectories from two tasks. "+reversal reward shaping" indicates training two separate potential models, one for successful trajectories from its own and the other for successful trajectories from the reversible task.
from two tasks, and (4) training two separate potential models, one for successful trajectories from its own and the other for successful trajectories from the reversible task. The final reward value is then computed as the average of the rewards obtained from these two models. Based on the ablation study shown Figure 20) we conclude that training two potential models always better than the baseline. Therefore, we use this setting to train potential models in subsequent experiments.
G Ablation Study of Potential Value Labeling Function in Potential-Base Reward Shaping
We evaluate four monotonically increasing functions as the potential value labeling function for successful trajectory of length n.
Linear: ((s))
Triangular: ((s:) ((t-1)
m(n+1))
Original Geometric: ((x)) = n-
Geometric: ((s:) n- 7--1
Results are shown in Figure with inter-quartile mean (IQM) Figure 22 Based on these results, adopt the linear function for potential-basee reward shaping in subsequent experiments.
25

---

Open
Door Open
0.4
lbsagska
Iycumers
etrinngt for
#gecmeencc
0.4
Adloode
Figure 21: Evaluation curves of agent success using time reversal symmetry guided reward shaping with different potential types.
0.-
Sangledast SAC
Vincar
00
Figure 22: IQM of agent success rate using time reversal symmetry guided reward shaping with different potential types.
Results of Time Reversal Symmetry Guided Reward Shaping in Robosuite H
As shown in Figure 23 we plot the evaluation curves of time reversal symmetry guided reward shaping in 6 environments of robosuite, from which we can see clear performance gap between the baseline and using time reversal symmetry guided reward shaping.
I Results of Both Proposed Techniques in Robosuite
Full evaluation curves of combining trajectory reversal augmentation with dynamics-aware filtering and time reversal symmetry guided reward shaping are provided in Figure 24] which confirm that combining both techniques yields superior performance compared to using either component alone or the baseline method.
Results of multi-task settings in Robosuite
Full evaluation curves of agent performance in 10 environments Robosuite are shown in Figure
26

---

Daer Ope1 rward
Open Dubward
3.-
02
04
cenernal
ceternaa maaic
Ducr Oles sward
Ortunid
Pog Serical
3-geriie 3Vc
027
230
135
350
Figure 23: Evaluation curves of time reversal symmetry guided reward shaping in 6 environments of robosuite. "Single-Taskk SAC" serves as the baseline. "+reversal reward shaping" introduces time reversal symmetry guided reward shaping.
K Additional results of MT50 in Meta-World
Additional results of agent performance in both 12 reversible task pairs and all 50 environments of MT50 are shown in Figure 26] Table 2
Figure 27 and Figure 28
L Compute Resources We Use
In all our experiments, utilize GPU server equipped with cards that have either RTX-4090 or A6000 GPUs and are powered by AMD EPYC 7763 CPUs. For experiments in robosuite: training a single-task agent takes around 5 hours while training multi-task agent for two tasks takes around 10 hours for 500 training episodes for each task. For experiments in MetaWorld: training a single-task agent takes around hours while training a multi-tasi agent for two tasks takes around 4 hours for 500 training episodes for each task. For MT50, it takes around 3 days to train an agent that handles 50 tasks.
M Limitations
key limitation of our work is the absence of real-robot experiments, as our current experiments are all conducted in simulation environments. While simulations enable efficient prototyping and scalability, they may oversimplify physical dynamics, sensor noise, or actuator constraints inherent in real-world robotic systems. Future work could address this gap by deploying the proposed method on physical robots, ensuring robustness and generalizability to practical applications. Like other data augmentation methods, our approach relies on prior knowledge of task structures specifically, time reversal symmetry in our case.
N Broader Impacts
Positive societal impacts This work advances the sample efficiency of deep reinforcement learning (DRL) agents in robotics manipulation tasks, enabling faster and more cost-effective training for practical applications. By reducing the computational resources required for training, it lowers barriers to deploying robotic systems in real-world setting. Improved sample efficiency also minimizes energy consumption hardware wear, aligning with sustainability goals. Furthermore, robust and efficient DRL. methodologies can accelerate the development of autonomous systems that enhance productivity, safety, and accessibility, ultimately contributing to economic growth and societal well-being.
27

---

Door Opan inward
Door Opan Outward
Insel Mearttioo
132
012
666
0.6
Eeplelet Sec.
SindeeTaal SAC
Sangl-Taa SAC
+nocerod zup
ereuersa a013
Door Close Inward
Ooor Close Outward
Plg Itemoval
1.0
1.
1.0
Segle
Single
02
022
enpornal
krsvernal
040
Siggle Intt 18.54C
00
Lingle leak 10 SMC
00
Singlo laak SaC
18C
2000
100
100
104
201
300
100
500
100
705
600
Block
ATONETE HDD
Teck
1.0
ATROECE resSId LT
08
14
ereversa revore
1OM GaacORS
18C
202
N6.1 Diseveerilly
Ricck Litd at
1.5
1.0
Singl-Ta
2.2
0.0
42 7ook th spr
stkk 10Ak boC
StEKk EOE SOC
64
SepleTao Th-54C
00
Sengh-Tes Taok TR.CAC
18C
300
400
302
25
22
26 5000
Namber balonds (K25)
Figure 24: Evaluation curves of both components in 10 environments of Robosuite. "reversal aug" represents incorporating reversal augmentation with filtering. "reversal reward shaping" represents incorporating potential based reward shaping. "Single-Tase Task TR-SAC" represents our proposed method which combines both components.
28

---

Opan
Open
inserton
Gengeebbe SAC
Se gle SAC
Se TegeeTost SAC pipleeiss
02
CadeCard 11d0C
04
Tadi-Card TixdiC
0.2
Tesi-Cand TiddG in iiieedd unt.
nutt sultinadd lot.
brieedd int.
most MAU
0.-
Head 5A0
Mi Head SA0
100
200
300
400
300
18c
200
300
400
500
332
700
300
400
500
apllodes
Door Closa Inward
Door Closa Outward
Plg) hemovaal
sggl TRAKC
21
0.5
0.6
No Hrid LAC
Skngelia TH-GG
Singleisa TH-AE
mndernnn san
Tol: Dwalt1 SXC
Tral Curd SSC
Sec
toad
0.
0.5
Nut Assemblyy
Block Stace
1001 Evail Success
Siigle
Tall-Cord SAC
Tal-Corr card TI-SAC
Mi LAC
aaplelbec TR-ANC
2.0
0.
NJt MECE 5AC
100
300
130
200
SOPN Frall
D'sasserribly
0.4
Skyle wA1
0/1
Sia for MAC
gio took Cord SAC
02
02
Task Card TB: Na b- b- SAC b--kc
Muts-ttedd SAC
Mt bitbieedd SAC
100
200
130
10
Figure 25: Evaluation curves for multi-task settings in 10 environments of Robosuite. "Task-Cond" and "Multi-Head" are short for "lask-conditioned" and "multi-headen respectively.
Negative societal impacts of our work.
To the best of our knowledge, we don't see any negative societal impacts
29

---

Number of success episodes reversible environments MT50
250
200
Number of success
150
100
Single-Task SAC Single-Task TR-SAC
Task-Cond SAC Task-Cond TR-SAC
Lang-Cond SAC
Lang-Cond TR-SAC
Figure 26: Average number of training episodes required to achieve a 100% success rate in 12 pairs of reversible environments of MT500..
Number of success episodes MT50
250
IOM Eval Success
Lang-Cord
Lang Cond TR.SAC
200
0.7
Number of success sooles
100
0.2
Number Fpisedes
Lang
Figure 27: IQM for agent success rate in all 50 environments of MT500
Figure 28: Average number of training episodes required achieve 100% success rate all 50 environments of MT50.
30

---

Table 2: Number of training episodes required for 100% success rate for 12 pairs of reversible environments in MT50. Each value is averaged over five runs, with the mean and standard deviation reported. "oaskCCond" and "Lang-Couu Lang-Cond" are short for "task-conditioned" and saaguagg-connitiooen respectively. Lower is better.
Single-Task SAC
Single-Tase Task TR-SAC
Task-Cond SAC
Task-Cond TR-SAC
Lang-Cond SAC
Lang-Cond TR-SAC
Environment
assembly
46027
432413
45618
424121
340:288
252:+29
disassemble
500++0
500:-0
500:1:0
500+
4244:21
364+24
coffee pull
444+116
408:-16
460:t4
468:19
416::24
416+:24
coffee push
288::16
2361:19
3161:24
224110
228:1:2
156:225
door lock
360+:26
3524:23
112#113
88::5
72+133
60+77
door unlock
196:122
1481:14
14417
88:19
96111 10
72:15
door open
252:+11
208+:16
208+:11
168+7
204
252:+-29
door close
100.14
8811
80:12
48
52:14
44:15
drawer open
184
200+
272::26
172::15
140=8
104+112
drawer close
40+3
16+11
16#2
12:11
8+:1
8+:1
faucet open
96.12
8411
92.15
80.13
48.15
40.16
faucet close
1241:
15647
72:12
60:23
76:22
64+5
handle press
32:14
20:1:2
20:1:2
24:13
24:11
20:10
handle pull
244-:14
1721:
308::200
128:17
168:124
160
peg insert side
328:1:22
296:118
348:118
276:116
424:1:11
348:::66
peg unplug side
164=113
12819
76
68+3
180#244
32:+1
plate slide
208++14
180+-14
1761-4
104+:11
80+4
64+77
plate slide back
256++18
224+:20
152++10
844:3
40+4
36+22
plate slide side
228:116
1561113
12819
148-120
80.17
60:16
plate slide back side
196+112
148+=4
140+:7
152+
104-9
72:t:5
push
288-117
140-1:0
272:1:24
2441188
404.127
328:1:00
push back
480.16
500.1O
316.1:2 22
396:1
368.1.44
324-130
window open
84:11
64+1
76:55
52:12
24:11
44::5
window close
80:12
68:11
64
40:10
56.1.5
52-122
ALL
2391:23
205::233
200:::24
1691:23
169::26
140:::55
31

---

Table 3: Number of training episodes required for 100% success rate for all 50 environments in MT50. Each value is averaged over five runs, with the mean and standard deviation reported. "Lang-Cond" "ondd short for
Lower is better.
Lang-Cond SAC
Lang-Cond TR-SAC
Lang-Cond SAC
Lang-Cond TR-SAC
Environment
Environment
assembly
340
252--29
sweep-into
276:27
172:224
disassemble
424+:21
364+:24
reach
104+4
164+:24
coffee pull
416:124
4161:24
reach wall
128:111
1884222
coffee push
228+:32
156++25
stick-pull
1724+23
160+-25
door lock
72+113
60+-7
sweep
316423
284+-25
door unlock
961110
72:15
basketball
500.10
50010
door open
204+-22
252:+29
bin picking
348:+26
360+:24
door close
52.14
44:15
box close
352.127
296.125
drawer open
140
104+:1
coffee button
52::3
64+7
drawer close
811
button press
28:122
36.12
faucet open
48:5
40:-6
button press wall
76
7646
faucet close
7612
64:15
button press topdown
220#:32
100+
button press
handle press
24
20:10
160::25
100+t
topdown wall
handle pull
168+:24
160+:25
dial turn
384-1:21
356+:25
handle pull side
304-:24
320+:31
handle press side
32:13
36+:2
peg insert side
424:1:1
348:126
hammer
172:1:23
160::255
peg unplug side
180+-24
32++1
hand insert
236++31
256++28
plate slide
8014
64:17
lever pull
280:1:27
252:1:30
plate slide back
40+4
36+-22
pick out of hole
424+-21
428++:00
plate slide side
8011
60:16
pick place
500-10
500.10
plate slide back side
104:-9
72:15
pick place wall
4164:44
416+:4
push
4044:27
328-:30
push wall
344+:28
3564:25
push back
368::24
324:130
shelf place
428:1:20
428:1:00
window open
24+1
44::5
soccer
3364:29
248+:29
window close
56.5
52:1-2
stick push
140.199
156416
Lang-Cond SAC
Lang Cond SAC
ALL
216=22
196=228
32


Reviews:
[{"review_id": "OuD6M8KDyW", "reviewer": "Reviewer_fKuw", "summary": "The paper presents Time Reversal Symmetry Enhanced Deep Reinforcement Learning to address challenges in robotic manipulation tasks within DRL. While previous methods primarily leverage spatial symmetries, this paper focuses on exploiting time reversal symmetry, considering that certain robot manipulations can be reversed in time, such as opening and closing a door or inserting and removing a peg. The results suggest that TR-DRL significantly enhances learning efficiency and final performance compared to baseline methods.", "questions": "1. Can the time reversal symmetry techniques be applied to a wider range of robotic tasks, or are they limited to tasks with explicitly reversible actions (e.g., peg insertion/removal)? It would be helpful if the authors could provide insight into how broadly these methods could be applied.\n\n2. Given that the paper does not include real-robot evaluations, how confident are the authors that their approach will perform well in a physical robot, particularly in environments with noise or variability in actuator performance? Future work could explore bridging this gap between simulation and real-world testing.\n\n3. In partially reversible tasks, how does the agent handle situations where only certain components of the state are reversible (e.g., door state vs. robot state)? A more detailed explanation of how partial reversibility impacts training would be useful.", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The introduction of time reversal symmetry in DRL is a novel and potentially impactful contribution. While spatial symmetries have been explored extensively, time reversal symmetry has not been fully leveraged in DRL.\n\n2. The two proposed techniques—trajectory reversal augmentation and time reversal guided reward shaping—are well thought out and methodically sound. The dynamics-consistent filter for ensuring valid reversed transitions is an excellent addition.\n\nWeaknesses:\n\n1. **Lack of Real-World Experiments**: While the paper provides extensive simulation-based experiments, the lack of real-world robotic evaluations is a key limitation. This is particularly relevant in robotics, where simulations often fail to capture real-world complexities such as sensor noise, physical dynamics, and actuator constraints.\n\n2. **Generalization to Other Domains**: The proposed methods are shown to work in robotics, but it is unclear how easily these techniques might generalize to other domains, such as those that do not have clear time reversal symmetry.\n\n3. **Complexity**: While the proposed methods are valuable, they introduce additional complexity into the reinforcement learning setup, such as the need for inverse dynamics models and filtering. The practical implementation of these components could be challenging in some settings.", "was_revised": true, "final_justification": "Resolved via rebuttal: The authors clarified scope and mechanics: distinction between FTR and PTR, the role of the dynamics-consistent filter, and why potential-based reward shaping is used to exploit partial reversibility without contaminating data. They also positioned the contribution as primarily improving the simulation training phase, which is reasonable.\n\nUnresolved: No real-robot evaluation or concrete sim-to-real strategy; limited robustness analysis under sensor/actuator noise and inverse-dynamics model errors; unclear generalization beyond tasks with clear time-reversal structure; added engineering complexity (inverse dynamics + filtering) not fully quantified.\n\nCamera-ready requests:\n\n1. Include at least a pilot real-robot or a strong sim-to-real study with domain randomization/noise ablations;\n\n2. Provide compute/complexity breakdown and sensitivity to inverse-dynamics inaccuracies;\n\n3. Broaden coverage of partially reversible tasks with diagnostics/guidelines for identifying TR symmetry.", "rebuttal": "Thank you for indicating our contributions and providing thoughtful feedback on our work. \nIn this work, we focus on robotics tasks with time reversal symmetry, and evaluate our proposed method in simulation. \nWe definitely agree that applying our approach in a real-world setting is an important step to prove the effectiveness of our method for solving those time-symmetric robotics tasks.\nAnd it is an interesting direction to learn the temporal symmetry if it does not obviously exist in tasks.\nWe leave the real-world experiments and learning temporal symmetry as future work.\nBelow, we answer each question in detail.\n\n**Q1**. \"Can the time reversal...\"\n\nWe would like to clarify that our method exploits two types of symmetry: full time reversal symmetry (FTR) and partial time reversal symmetry (PTR). \nPeg insertion/removal and door opening/closing inward are examples of FTR and PTR task pairs respectively.\nIn door opening/closing inward, most reversed transitions are filtered out by the dynamics-consistent filter, since the agent cannot reverse the action (i.e., from \"push the door\" to \"grasp the handle and pull the door\") without first grasping the handle.\nThe detailed examples of partial reversibility can be found in Example 2 of Section 4.1.\nWe design a time reversal reward shaping mechanism to handle these partially reversible transitions.\nA potential model is trained with successful trajectories of task A and used to guide the agent training of task B through potential-based reward shaping, as detailed in Section 4.3.\n\n**Q2**. \"Given that the paper...\"\n\nAs stated at the beginning of the response, we admit the significance of real-world experiments and leave it as future work.\nConsidering the difficulties of conducting RL directly under real-world setting, a common practice is to train the policy with RL in simulation and deploy the policy in real-world by solving the sim-to-real gap.\nOur approach can effectively facilitate the learning process during the simulation phase.\n\n**Q3**. \"In partially reversible tasks...\"\n\nFor tasks where only subsets of states exhibit time reversal symmetry, reversed trajectories obtained from trajectory reversal augmentation often violate true environment dynamics, leading to low quality data that may even hurt agent training.\nInstead, our method leverages time reversal guided reward shaping to exploit the partial reversibility.\nIntuitively, for a high-reward trajectory from one task (door from open to closed), if a trajectory from another task can achieve the reversed object states (door from closed to open), it should likewise receive a high reward.\nThis motivates us to employ reward shaping to guide the RL agent with these partially reversible transitions, as detailed in Section 4.3.\nFurthermore, we choose to use potential-based reward shaping as it preserves policy optimality and train a potential model so that we can fully utilize all successful trajectories."}, {"review_id": "kZTEd4Po2H", "reviewer": "Reviewer_TdWt", "summary": "This paper introduces Time Reversal Deep RL (TD-DRL), a framework for exploring time-reversal symmetry in robotics tasks to improve the data efficiency of RL. TR-DRL generates additional augmented data that respect the dynamics of the task by reversing trajectories, and then incorporates this data into off-policy RL algorithms. Empirically, TR-DRL improves data efficiency on Robosuite and Metaworld tasks.", "questions": "1. \"most reversed transitions are filtered out by the dynamics-consistent filter\" Could you support this empirically? It'd be interesting to see  how much is filtered out.\n1. \"The training process alternates between two reversible tasks.\" Is this a critical part of the algorithm, or was this just chosen for convenience?\n1. Do the authors have any sense of how the augmented data compares to additional policy-generated data? In particular, suppose we replaced the augmented data with additional data collected by the agent (and suppose we give the agent this data \"for free\" in that we don't count this data as additional timesteps).", "limitations": "Yes, but I think it's worth mentioning that this method assumes we know the task has time reversibility. I don't think this is a significant shortcoming though, since all data augmentation papers assume knowledge of some function that can transform observed data into additional training data (e.g. an augmentation function). This is just an inherent limitation of data augmentation methods. Corrado & Hanna [1] and Pitis et al. [2] discuss this point.\n\n1. Corrado & Hanna. \"Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates.\" https://arxiv.org/abs/2310.17786\n3. Pitis et al. \"Counterfactual Data Augmentation using Locally Factored Dynamics.\"  https://arxiv.org/abs/2007.02863", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n1. Partial time reversal symmetry appears in many real world robotics, so the topic is relevant and practically useful. \n2. TR-DRL outperforms baselines by a fair margin in all tasks consider.\n3. From the ablation studies, I gather that TR-DRL is fairly robust to hyperparameter choices and that each component of the method helps improve performance. \n\n# Weaknesses\n\n1. **Baselines.** TR-DRL is the only method that leverages this knowledge and thus naturally has an advantage over vanilla baselines. I wouldn't say this is necessarily an unfair advantage, though; a core contribution of this paper is showing the benefit of incorporating knowledge of time reversal into training. However, I think some discussion is warranted on other ways we could potentially leverage this information. e.g. learning reversible action representations [1]. \nTo rephrase my comment: Experiments show that adding time-reversed augmentated data to the replay buffer improves data efficiency. Are there other ways to integrate time-reversal information? How do they compare?\n\n1. **Time reversal examples are a bit unclear.** My understanding other partial time reversal is rooted in my understanding of locally factored dynamics discussed by Pitis et. al. [2] (see the next section below for additional context). I had a bit of trouble understanding what Example 2 was talking about until I made this connection. Are we saying that the door state is reversible because there exists some action that can reverse the door's trajectory?\n\n1. **Mild impact.** The paper addresses a very specific problem that would be primarily of interest to robotics-focused practitioners rather than the broader RL community. \n\n# Other comments\n\n1. The definition of partial time reversibility (line 149) reminds me of the definition of \"locally factored dynamics\" discussed in CoDA by Pitis et. al. [2]. Locally factored dynamics means that after taking an action, only a small number of components of the next state s’ change. For instance, an object won't move unless the agent touches it, so if the agent and object are physically separated, the agent's dynamics are completely independent of the object's dynamics. This local structure allows Pitis et al. [2] to generate counterfactual transitions by swapping parts of next states across trajectories that share common context in the unaffected dimensions. You may be able to leverage ideas/definitions from [2] to motivate partial time reversibility. Followup works MoCoDA [3] and RoCoDA [4] may be of interest. \n\n# Minor Comments\n\n1. \"existing work (see Related Work in Section 2) predominantly focuses on spatial symmetries, 28 such as translation, reflection, and rotation...\" The authors might also mention augmentation methods that aim to improve robustness to noise (i.e. essentially all visual augmentation works, S4RL [5], and \"sample enhancement\" in [6])\n2. Line 33: \"speed of action execution\" in this context, speed refers to the frequency at which actions are selected, right?\n3. Figure 2: I should think of the reversal of this trajectories as reading the images in reverse from right to left, correct? Also, I suggest writing \"Figure 2: Examples of reversible and irreversible trajectories\" would help clarify what this figure is showing. (a) Reversible: ... (b) Irreversible ...\" The juxtaposition of (a) and (b) made me think (b) was meant to represent the reversal of (a).\n4. Line 105: the definitions of the dynamics function suggests stochastic dynamics, but does stochasticity interfere with reversibility? \n1. Line 111: The overhead arrow notation is not explicitly defined prior ot using it.\n1. Line 115: I think it's worth emphasizes somewhere that these sorts of symmetries frequently arise in the real world, so it's reasonable to assume these involutions exist.\n1. When defining inverse dynamics models, it would help to unify notation a bit. For instance, $f_a, f_s, $ could denote dynamics for \n1. Figures 5 and 6 would be much clearer if the results for each algorithm pair (TR vs. no TR)were plotted separately. \n1. Line 115 \"iin\" typo\n---\n# References  \n\n1. Xie et al. \"Learning Latent Representations to Influence Multi-Agent Interaction.\" https://arxiv.org/abs/2011.06619\n3. Pitis et al. \"Counterfactual Data Augmentation using Locally Factored Dynamics.\"  https://arxiv.org/abs/2007.02863\n4. Pitis et al. MoCoDA: Model-based Counterfactual Data Augmentation. https://arxiv.org/abs/2007.02863https://arxiv.org/abs/2210.11287\n4. Ameperosa et al. \"RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrations.\" https://arxiv.org/abs/2411.16959 \n1. Sinha et al. S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning. https://arxiv.org/abs/2103.06326\n2. Qiao et al. \"Efficient Differentiable Simulation of Articulated Bodies.\" https://arxiv.org/abs/2109.07719", "was_revised": true, "final_justification": "The paper is properly motivated, technically correct, and adequately supports stated claims. My comments were mostly clarification questions and can easily be addressed by minor changes to a camera-ready version.\n\nI do want to emphasize that because the method is of most interest to roboticists but is not evaluated on physical robotics tasks, impact is limited. However, I put much more weight in the \"correctness\" and \"relevance\" aspects of research, so I do no seet this drawback as a reason to reject given the work's other merits.", "rebuttal": "Thank you for summarizing our work and providing thoughtful feedback that helps us strengthen both the presentation and technical depth of our paper.\nBelow we answer your questions in detail.\n\n**Weakness1**. \"Baselines...\"\n\nAs detailed in related works (Section 2), there are several other ways to integrate time-reversal information.\n\nSome existing works exploit reversibility from goal states to enhance exploration of the agents, similar to what we would like to achieve through the time reversal guided reward shaping technique.\nOur method differs from them in that our proposed trajectory reversal augmentation technique leverages time reversal symmetry not only from goal states but globally for every transition in the trajectory, enabling a broader application of time symmetry across the entire state space.\n\nOther prior works are designed to avoid irreversible actions of the agent for safer exploration.\nOne example is \"learning a reset policy alongside the normal policy to prevent agents from entering non-reversible states, ensuring safety in exploration phase and achieving better training efficiency\".\nUnlike their reset policy that sets the initial state as the ending state of the current policy and the goal state as the task's starting point, our method treats two reversible tasks independently, with initial and goal states defined separately for each task.\nMoreover, our method is orthogonal to theirs and can be integrated to enhance the training of their reset policy.\n\n**Weakness2**. \"Time reversal examples...\"\n\nWe have updated our manuscript to recall state decomposition proposed by this work when we introduce the example of partial time reversal symmetry.\nThe answer to your question in **Weakness2** is: Yes. \nBy our definition in Section 4.1, partial time reversal symmetry indicates that a subset of the whole state is reversible. \nIn the example of door opening/closing inward, it means that the door's trajectory can be reversed by some action.\n\n**Minor1**. \"existing work...\"\n\nThank you for highlighting this important connection. \nWe have expanded the related work to explicitly include how data augmentation methods improve noise robustness and cited these two works as key examples.\n\n**Minor2**. \"Line 33:...\"\n\nYes, \"the speed of action execution\" is the frequency at which actions are selected. \nThis is often controlled by a hyperparameter called \"action repeat\" in RL methods. \nA larger value of \"action repeat\" indicates a lower control frequency.\n\n**Minor3**. \"Figure 2:...\"\n\nThanks for this constructive suggestion. \nWe have updated our manuscript accordingly.\n\n**Minor4**. \"Line 105:...\"\n\nAs defined in line 110, time reversal symmetry applies to probability distributions over transitions. \nConsequently, stochastic dynamics does not interfere with reversibility, as it can remain statistically consistent under time reversal.\n\n**Minor5**. \"Line 111:...\"\n\nTo avoid confusion, we have explicitly defined the overhead arrow notation as time reversal operation on $\\mathcal{S}$ or $\\mathcal{A}$ at line 110 of our manuscript.\n\n**Minor6**. \"Line 115:...\"\n\nThank you for highlighting this important point. \nAs noted by Barkley et al. (cited at line 110), such time symmetries commonly emerge in physical systems. \nWe have now emphasized this point explicitly near line 115 in our manuscript.\n\n**Minor7**. \"When defining...\"\n\nBoth $f_{\\mathcal S}$ and $f_{\\mathcal A}$ are used in Section 3 for involutions over states and actions.\nWe choose to stick with the current notations for inverse and forward dynamics models ($h$ and $g$) to avoid misunderstanding.\n\n**Minor8**. \"Figures 5 and 6...\"\n\nDue to page limit in the main text, we combine the results of all methods into a single plot. \nWe will include separate plots for each algorithm pair (TR vs. no TR) in the appendix for better readability.\n\n**Question1**. \"most reversed transitions...\"\n\nWe empirically validate this point through quantitative analysis. \nWe have tested our filtering method on the agent's saved replay buffer, which confirms that only 27.27\\% of transitions in door closing inward task pass filtering versus 63.13\\% for door closing outward task.\nSpecifically, when closing a door inward, transitions of the gripper approaching the door can be reverted, while the transitions of the gripper pushing the door fail to pass filtering due to their violations of learned environment dynamics.\n\n**Question2**. \"The training process alternates...\"\n\nWe would like to clarify that data collection (agent interacting with environments) in our method alternates between two tasks purely for implementation convenience.\nIt is expected to be more efficient to interact with the two environments in parallel.\nDuring agent update step, we sample transitions from replay buffers of both environments simultaneously, and update all models with these transitions.\n\n**Question3**. \"Do the authors...\"\n\nTime reversal augmented data differs from policy generated data in quality and utility.\nOne advantage of policy generated data is that it is 100\\% realistic and obeys environment dynamics, while time reversal augmented data, even after filtering, may not be fully dynamics-consistent.\nConversely, there are two advantages of time reversal augmented data.\nFirst, it could help improving agent exploration at the beginning of the training, since the trajectories from two tasks cover distinct regions of the state space.\nThis synergy is validated by the results shown in Figure 22 of our manuscript, where doubling the amount of policy generated data can not reach the performance of time reversal augmented data in this case.\nSecond, the agent may converge at different speeds on the two tasks.\nA successful trajectory from one task can effectively guide the training for the other.\n\n**Limitations**. \"Yes, but I think it's worth mentioning...\"\n\nWe fully agree with this point and have explicitly included the following sentence in the limitations section of our manuscript. \n\"Like other data augmentation methods, our approach relies on prior knowledge of task structures—specifically, time reversal symmetry in our case.\""}, {"review_id": "x67sAHgnNe", "reviewer": "Reviewer_Pawe", "summary": "In this work, the authors present a simple yet effective trick for data augmentation in reinforcement learning methods using temporal symmetry that they call \"Time Reversal Symmetry\" augmentation. This augmentation take advantage of the following facts (i) often in a replay buffer, there exist reversible state transitions (ii) these reversible transitions can be identified with learned dynamics models (iii) once identified, they can be used to create a potential function to augment the reward. Once used properly, this method is able to increase the data efficiency of reinforcement learning methods, as shown in experiments in MetaWorld and RoboSuite environments.", "questions": "1. While the work only focuses on involutions created by forward and inverse dynamics models, could the authors think of some more interesting, maybe simpler involutions, which could take this beyond simple \"negate the action sign\" model? Perhaps even a learned action inverting model could give interesting results.\n2. What would the results look like without the guiding demonstrations seeding the replay buffer? Would this method make a difference when exploring tabula rasa? If no, why not?", "limitations": "While the authors mention the lack of real world experiments, there are further limitations beyond that. Particularly, the method currently is limited to quasi-static environments, and would be hard to extend to domains such as video games or self-driving because the environments are generally dynamic and there is a definite \"forward arrow of time\" in such settings.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "This work in particular is impressive in that it takes a simple idea, and with proper evaluation, shows that it leads to improvement in some real benchmark tasks. The idea is sensible, and the method design that leads around the idea feels like the \"shortest path\" needed to reach the effective method, which is desirable and yet rare in recent works. The ablation studies presented in the paper are also sensible, and does the ablations at the right level of abstraction to prove the necessity of each of the pieces.\n\nWhile the idea behind the work and the presented method are strong, the benchmarking and the results leave something to be desired. Particularly, evaluating on two somewhat simple simulated environments with some demonstration data already in the buffer makes the tasks accomplished incredibly simple. At the same time, from the plots, it seems that the performance improvements are not always clear – often there are significant overlaps in the error bars, and some more rigorous evaluations with more seeds, or more thorough investigation on a larger set of simulated and real environments, while not critical at this stage, will make this work significantly stronger.", "was_revised": true, "final_justification": "I appreciate the updates from the authors, and have therefore updated my confidence in my score. With some real robot experiments it would be a strong accept, but this work is valuable even without such experiments and justifies my \"Accept\" rating.", "rebuttal": "Thank you for noting the quality and clarity of our work. \nWe are happy that you recognize that the model design leading around our idea feels like the \"shortest path\". \nBelow, we address each question thoroughly.\n\n**Q1**. \"While the work...\"\n\nWe need to clarify that our method specifically learns an 'action inverting model' implemented via an inverse dynamics model.\nUnlike Barkley et al.'s approach [1] of simply negating actions in reversed transitions—which is not always valid—our model derives reverted actions through a learned inverse dynamics.\nHowever, for transitions without full time reversal symmetry, the inverse dynamics model may still generate invalid actions.\nTo address this, we introduce a forward dynamics model to filter out such invalid transitions.        \n\n**Q2**. \"What would the results...\"\n\nIn the sparse-reward setting, exploring tabula rasa is a very challenging task for the RL agent.\nConsequently, agent performance without demonstrations drops substantially, which is a key reason that we do not include these experiments in the manuscript.\nOn challenging tasks such as door opening/closing outward, nut assembly/disassembly, and block stacking/unstacking, the agents fail to make progress under sparse rewards alone. \nHowever, on other tasks like door opening/closing inward and peg insertion/removal, the IQM of success rate (shown in the tables below) reveal that the performance gap between TR-DRL and baseline methods widens significantly without demonstrations, confirming that time reversal symmetry provides guidance and partially compensates for missing demonstration data.\n        \n \"Single-Task SAC\": baseline; \"+reversal aug\": trajectory reversal augmentation with dynamics-aware filtering; \"+reversal reward shaping\": time reversal symmetry guided reward shaping.\n|Task|Number of Transitions |50k|100k|150k|200k|\n|:----------------- | :-------------------------------- | :------ | :------ | :------ | :------ |\n|**Door opening inward**||||||\n|Single-Task SAC|  | `0.00±0.00`  | `0.00±0.00`  | `0.00±0.00`  | `0.00±0.00`  |\n|+ reversal aug| | `0.00±0.00`  | `0.00±0.00`  | `0.00±0.00`  | `0.18±0.36`  |\n|+ reversal reward shaping || `0.00±0.00`  | **`0.13±0.26`** | **`0.31±0.41`** | `0.60±0.38`  |\n|Single-Task TR-SAC **(Ours)**|| `0.00±0.00`  | `0.11±0.22`  | `0.26±0.37`  | **`0.72±0.37`** |\n|**Door closing inward**||||||\n|Single-Task SAC|  | `0.00±0.00`  | `0.00±0.00`  | `0.05±0.09`  | `0.19±0.36`  |\n|+ reversal aug| | `0.00±0.00`  | `0.20±0.40`  | `0.20±0.40`  | `0.37±0.45`  |\n|+ reversal reward shaping || `0.03±0.06`  | **`0.29±0.28`** | `0.46±0.38` | `0.69±0.35`  |\n|Single-Task TR-SAC **(Ours)**|| `0.02±0.03`  | `0.29±0.35`  | **`0.55±0.41`**  | **`0.77±0.30`** |\n|**Peg Insertion**||||||\n|Single-Task SAC|  | `0.00±0.00`  | `0.00±0.00`  | `0.00±0.00`  | `0.23±0.24`  |\n|+ reversal aug| | `0.05±0.09`  | `0.37±0.33`  | `0.58±0.37`  | `0.95±0.07`  |\n|+ reversal reward shaping || `0.07±0.09`  | `0.37±0.44` | `0.55±0.45` | `0.75±0.38`  |\n|Single-Task TR-SAC **(Ours)**|| **`0.16±0.16`**  | **`0.50±0.39`**  | **`0.91±0.07`**  | **`1.00±0.00`** |\n|**Peg Removal**||||||\n|Single-Task SAC|  | `0.00±0.00`  | `0.09±0.17`  | `0.27±0.39`  | `0.64±0.37`  |\n|+ reversal aug| | **`0.25±0.26`**  | `0.81±0.31`  | **`1.00±0.00`**  | **`1.00±0.00`**  |\n|+ reversal reward shaping || `0.00±0.00`  | `0.23±0.39` | `0.52±0.44` | `0.63±0.45`  |\n|Single-Task TR-SAC **(Ours)**|| `0.08±0.10`  | **`0.86±0.15`**  | **`1.00±0.00`**  | **`1.00±0.00`** |\n\n[1]. Barkley et al. \"An Investigation of Time Reversal Symmetry in Reinforcement Learning.\" https://arxiv.org/abs/2311.17008"}, {"review_id": "joTWaFZEqv", "reviewer": "Reviewer_dzfv", "summary": "The paper explores an intuitive approach to improving the sample efficiency of RL by using a time reversal symmetry. Time reversal symmetry is when an experience can be played backward in time in order to get information about how to perform the reversed version of a task, e.g. cloing a drawer versus opening a drawer. They explore two different methods. In the first (full time reversal), the method identifies those transitions that are fully reversible by training a forward and inverse dynamics model and evaluating the degree of match between them. Transitions that are fully reversible are reversed and added to the replay buffer. In the second method (partial time reveral), the portion of state that is reversible is used to guide RL exploration via a potential function that provides a dense reward signal. The paper presents compelling robosuite experiments that demonstrate that the method is helpful in most situations. Fig 6 is particularly compelling.", "questions": "It was all pretty clear to me.", "limitations": "See my weaknesses above. I think complexity is a limitation. I also think that there there might be more effective ways to do this beyond data augmentation and dense rewardifying.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ I liked the main idea of the paper. I liked that it was intuitive. This seems like an approach that should be helpful in a variety of situations.\n\n+ I thought that the methods that were proposed were pretty reasonable and the experimental results indicate that they are generally helpful.\n\n+ I thought the experiments were good. I was convinced.\n\n- A key question here is whether this method passes the bang-for-buck threshold. In order to use this method, you need to learn forward and inverse dynamics models, significantly adding to the compute complexity of the alg. There is a benefit to doing so, but it may not be enough on its own to warrant learning those extra models. On the other hand, there are probably plenty of situations where folks are already learning those dynamics models and the method would definitely make sense there.\n\n- I can't help but wonder if there might be more effective and sophisticated ways of leveraging the time reversal symmetry. There should be some systems for which time reversal should yield a 2x speedup. Are there ways to maybe incorporate time reveral directly into the neural model? Not clear, but interesting.", "was_revised": true, "final_justification": "I think this is a solid paper that should be accepted if possible. I like the intuition of the idea of time reversal symmetry. My main criticisms are: 1) I suspect there are more sophisticated ways of leveraging this intuition; 2) the current method is a little \"heavy\" in terms of requiring learning of additional forward and inverse models. Re the second criticism, as the reviewers point out in the rebuttal, this is probably not a huge ask since it is not uncommon to do this.", "rebuttal": "We sincerely appreciate your positive evaluation, especially noting the intuitiveness of our approach.\nIn response to your observations about weaknesses, below we provide answers in detail.\n\n**Weakness1**. \"A key question here...\"\n\nWe agree that learning forward and inverse dynamics models has additional computational cost but emphasize that superior sample efficiency is a critical goal in deep RL. \nLike common techniques such as data augmentation or auxiliary losses, this added computation directly enables efficient learning with fewer samples and better final performance. \nFurthermore, our method seamlessly integrates with existing approaches that have already trained dynamics models, offering enhanced performance without incremental computational cost in such cases.\n\n**Weakness2**. \"I can't help but wonder...\"\n\nWe appreciate your insightful suggestion about sophisticated ways of leveraging time reversal symmetry and we agree that it is an interesting topic.\nOne possible approach is to incorporate the temporal symmetry into the neural network architecture design, similar to steerable CNNs handling spatial symmetry like rotation equivariance.\nMoreover, we suspect that the 2x speedup may be too optimistic.\nThis would require idealized conditions rarely met in practice, including perfectly reversible dynamics and accurate backward policy execution."}]

Author rebuttal:

Meta-review: The paper is in the line of research where symmetries in robot motions are used to make robot manipulation more efficient when using reinforcement learning for optimization. This paper focuses on time reversal symmetries where the idea is that for example a robot can sometimes reverse its actions, for example, walk forwards, and then walk back backwards. Previously, other kinds of symmetries such as symmetric shapes of objects have been used.

A strength of the paper is the introduction of the time reversal symmetries to robot reinforcement learning which is a strong contribution. While the contribution is at the moment limited to the robotics setting, it is of sufficient impact.

A weakness of the paper is lack of real robot experiments, or, sim2sim experiments with noise or other kind of perturbations to the environment. However, even without real robot experiments the paper provides an interesting contribution.

During the rebuttal the authors added further experimental results and answered satisfactorily reviewer questions. Also the applicability to other domains was discussed. This was left out of the scope of the paper.
Decision: Accept (poster)