Title: Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deep Reinforcement Learning

Robotics Control
Abstract: Symmetry is pervasive in robotics and has been widely exploited to improve sample efficiency in deep reinforcement learning (DRL). However, existing approaches primarily focus on spatial symmetries—such as reflection, rotation, and translation—while largely neglecting temporal symmetries. To address this gap, we explore time reversal symmetry, a form of temporal symmetry commonly found in robotics tasks such as door opening and closing. We propose Time Reversal symmetry enhanced Deep Reinforcement Learning (TR-DRL), a framework that combines trajectory reversal augmentation and time reversal guided reward shaping to efficiently solve temporally symmetric tasks. Our method generates reversed transitions from fully reversible transitions, identified by a proposed dynamics-consistent filter, to augment the training data. For partially reversible transitions, we apply reward shaping to guide learning, according to successful trajectories from the reversed task. Extensive experiments on the Robosuite and MetaWorld benchmarks demonstrate that TR-DRL is effective in both single-task and multi-task settings, achieving higher sample efficiency and stronger final performance compared to baseline methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores an intuitive approach to improving the sample efficiency of RL by using a time reversal symmetry. Time reversal symmetry is when an experience can be played backward in time in order to get information about how to perform the reversed version of a task, e.g. cloing a drawer versus opening a drawer. They explore two different methods. In the first (full time reversal), the method identifies those transitions that are fully reversible by training a forward and inverse dynamics model and evaluating the degree of match between them. Transitions that are fully reversible are reversed and added to the replay buffer. In the second method (partial time reveral), the portion of state that is reversible is used to guide RL exploration via a potential function that provides a dense reward signal. The paper presents compelling robosuite experiments that demonstrate that the method is helpful in most situations. Fig 6 is particularly compelling.\n\nSTRENGTHS AND WEAKNESSES:\n+ I liked the main idea of the paper. I liked that it was intuitive. This seems like an approach that should be helpful in a variety of situations.\n\n+ I thought that the methods that were proposed were pretty reasonable and the experimental results indicate that they are generally helpful.\n\n+ I thought the experiments were good. I was convinced.\n\n- A key question here is whether this method passes the bang-for-buck threshold. In order to use this method, you need to learn forward and inverse dynamics models, significantly adding to the compute complexity of the alg. There is a benefit to doing so, but it may not be enough on its own to warrant learning those extra models. On the other hand, there are probably plenty of situations where folks are already learning those dynamics models and the method would definitely make sense there.\n\n- I can't help but wonder if there might be more effective and sophisticated ways of leveraging the time reversal symmetry. There should be some systems for which time reversal should yield a 2x speedup. Are there ways to maybe incorporate time reveral directly into the neural model? Not clear, but interesting.\n\nQUESTIONS:\nIt was all pretty clear to me.\n\nLIMITATIONS:\nSee my weaknesses above. I think complexity is a limitation. I also think that there there might be more effective ways to do this beyond data augmentation and dense rewardifying.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present a simple yet effective trick for data augmentation in reinforcement learning methods using temporal symmetry that they call \"Time Reversal Symmetry\" augmentation. This augmentation take advantage of the following facts (i) often in a replay buffer, there exist reversible state transitions (ii) these reversible transitions can be identified with learned dynamics models (iii) once identified, they can be used to create a potential function to augment the reward. Once used properly, this method is able to increase the data efficiency of reinforcement learning methods, as shown in experiments in MetaWorld and RoboSuite environments.\n\nSTRENGTHS AND WEAKNESSES:\nThis work in particular is impressive in that it takes a simple idea, and with proper evaluation, shows that it leads to improvement in some real benchmark tasks. The idea is sensible, and the method design that leads around the idea feels like the \"shortest path\" needed to reach the effective method, which is desirable and yet rare in recent works. The ablation studies presented in the paper are also sensible, and does the ablations at the right level of abstraction to prove the necessity of each of the pieces.\n\nWhile the idea behind the work and the presented method are strong, the benchmarking and the results leave something to be desired. Particularly, evaluating on two somewhat simple simulated environments with some demonstration data already in the buffer makes the tasks accomplished incredibly simple. At the same time, from the plots, it seems that the performance improvements are not always clear – often there are significant overlaps in the error bars, and some more rigorous evaluations with more seeds, or more thorough investigation on a larger set of simulated and real environments, while not critical at this stage, will make this work significantly stronger.\n\nQUESTIONS:\n1. While the work only focuses on involutions created by forward and inverse dynamics models, could the authors think of some more interesting, maybe simpler involutions, which could take this beyond simple \"negate the action sign\" model? Perhaps even a learned action inverting model could give interesting results.\n2. What would the results look like without the guiding demonstrations seeding the replay buffer? Would this method make a difference when exploring tabula rasa? If no, why not?\n\nLIMITATIONS:\nWhile the authors mention the lack of real world experiments, there are further limitations beyond that. Particularly, the method currently is limited to quasi-static environments, and would be hard to extend to domains such as video games or self-driving because the environments are generally dynamic and there is a definite \"forward arrow of time\" in such settings.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Time Reversal Deep RL (TD-DRL), a framework for exploring time-reversal symmetry in robotics tasks to improve the data efficiency of RL. TR-DRL generates additional augmented data that respect the dynamics of the task by reversing trajectories, and then incorporates this data into off-policy RL algorithms. Empirically, TR-DRL improves data efficiency on Robosuite and Metaworld tasks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. Partial time reversal symmetry appears in many real world robotics, so the topic is relevant and practically useful. \n2. TR-DRL outperforms baselines by a fair margin in all tasks consider.\n3. From the ablation studies, I gather that TR-DRL is fairly robust to hyperparameter choices and that each component of the method helps improve performance. \n\n# Weaknesses\n\n1. **Baselines.** TR-DRL is the only method that leverages this knowledge and thus naturally has an advantage over vanilla baselines. I wouldn't say this is necessarily an unfair advantage, though; a core contribution of this paper is showing the benefit of incorporating knowledge of time reversal into training. However, I think some discussion is warranted on other ways we could potentially leverage this information. e.g. learning reversible action representations [1]. \nTo rephrase my comment: Experiments show that adding time-reversed augmentated data to the replay buffer improves data efficiency. Are there other ways to integrate time-reversal information? How do they compare?\n\n1. **Time reversal examples are a bit unclear.** My understanding other partial time reversal is rooted in my understanding of locally factored dynamics discussed by Pitis et. al. [2] (see the next section below for additional context). I had a bit of trouble understanding what Example 2 was talking about until I made this connection. Are we saying that the door state is reversible because there exists some action that can reverse the door's trajectory?\n\n1. **Mild impact.** The paper addresses a very specific problem that would be primarily of interest to robotics-focused practitioners rather than the broader RL community. \n\n# Other comments\n\n1. The definition of partial time reversibility (line 149) reminds me of the definition of \"locally factored dynamics\" discussed in CoDA by Pitis et. al. [2]. Locally factored dynamics means that after taking an action, only a small number of components of the next state s’ change. For instance, an object won't move unless the agent touches it, so if the agent and object are physically separated, the agent's dynamics are completely independent of the object's dynamics. This local structure allows Pitis et al. [2] to generate counterfactual transitions by swapping parts of next states across trajectories that share common context in the unaffected dimensions. You may be able to leverage ideas/definitions from [2] to motivate partial time reversibility. Followup works MoCoDA [3] and RoCoDA [4] may be of interest. \n\n# Minor Comments\n\n1. \"existing work (see Related Work in Section 2) predominantly focuses on spatial symmetries, 28 such as translation, reflection, and rotation...\" The authors might also mention augmentation methods that aim to improve robustness to noise (i.e. essentially all visual augmentation works, S4RL [5], and \"sample enhancement\" in [6])\n2. Line 33: \"speed of action execution\" in this context, speed refers to the frequency at which actions are selected, right?\n3. Figure 2: I should think of the reversal of this trajectories as reading the images in reverse from right to left, correct? Also, I suggest writing \"Figure 2: Examples of reversible and irreversible trajectories\" would help clarify what this figure is showing. (a) Reversible: ... (b) Irreversible ...\" The juxtaposition of (a) and (b) made me think (b) was meant to represent the reversal of (a).\n4. Line 105: the definitions of the dynamics function suggests stochastic dynamics, but does stochasticity interfere with reversibility? \n1. Line 111: The overhead arrow notation is not explicitly defined prior ot using it.\n1. Line 115: I think it's worth emphasizes somewhere that these sorts of symmetries frequently arise in the real world, so it's reasonable to assume these involutions exist.\n1. When defining inverse dynamics models, it would help to unify notation a bit. For instance, $f_a, f_s, $ could denote dynamics for \n1. Figures 5 and 6 would be much clearer if the results for each algorithm pair (TR vs. no TR)were plotted separately. \n1. Line 115 \"iin\" typo\n---\n# References  \n\n1. Xie et al. \"Learning Latent Representations to Influence Multi-Agent Interaction.\" https://arxiv.org/abs/2011.06619\n3. Pitis et al. \"Counterfactual Data Augmentation using Locally Factored Dynamics.\"  https://arxiv.org/abs/2007.02863\n4. Pitis et al. MoCoDA: Model-based Counterfactual Data Augmentation. https://arxiv.org/abs/2007.02863https://arxiv.org/abs/2210.11287\n4. Ameperosa et al. \"RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrations.\" https://arxiv.org/abs/2411.16959 \n1. Sinha et al. S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning. https://arxiv.org/abs/2103.06326\n2. Qiao et al. \"Efficient Differentiable Simulation of Articulated Bodies.\" https://arxiv.org/abs/2109.07719\n\nQUESTIONS:\n1. \"most reversed transitions are filtered out by the dynamics-consistent filter\" Could you support this empirically? It'd be interesting to see  how much is filtered out.\n1. \"The training process alternates between two reversible tasks.\" Is this a critical part of the algorithm, or was this just chosen for convenience?\n1. Do the authors have any sense of how the augmented data compares to additional policy-generated data? In particular, suppose we replaced the augmented data with additional data collected by the agent (and suppose we give the agent this data \"for free\" in that we don't count this data as additional timesteps).\n\nLIMITATIONS:\nYes, but I think it's worth mentioning that this method assumes we know the task has time reversibility. I don't think this is a significant shortcoming though, since all data augmentation papers assume knowledge of some function that can transform observed data into additional training data (e.g. an augmentation function). This is just an inherent limitation of data augmentation methods. Corrado & Hanna [1] and Pitis et al. [2] discuss this point.\n\n1. Corrado & Hanna. \"Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates.\" https://arxiv.org/abs/2310.17786\n3. Pitis et al. \"Counterfactual Data Augmentation using Locally Factored Dynamics.\"  https://arxiv.org/abs/2007.02863", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Time Reversal Symmetry Enhanced Deep Reinforcement Learning to address challenges in robotic manipulation tasks within DRL. While previous methods primarily leverage spatial symmetries, this paper focuses on exploiting time reversal symmetry, considering that certain robot manipulations can be reversed in time, such as opening and closing a door or inserting and removing a peg. The results suggest that TR-DRL significantly enhances learning efficiency and final performance compared to baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The introduction of time reversal symmetry in DRL is a novel and potentially impactful contribution. While spatial symmetries have been explored extensively, time reversal symmetry has not been fully leveraged in DRL.\n\n2. The two proposed techniques—trajectory reversal augmentation and time reversal guided reward shaping—are well thought out and methodically sound. The dynamics-consistent filter for ensuring valid reversed transitions is an excellent addition.\n\nWeaknesses:\n\n1. **Lack of Real-World Experiments**: While the paper provides extensive simulation-based experiments, the lack of real-world robotic evaluations is a key limitation. This is particularly relevant in robotics, where simulations often fail to capture real-world complexities such as sensor noise, physical dynamics, and actuator constraints.\n\n2. **Generalization to Other Domains**: The proposed methods are shown to work in robotics, but it is unclear how easily these techniques might generalize to other domains, such as those that do not have clear time reversal symmetry.\n\n3. **Complexity**: While the proposed methods are valuable, they introduce additional complexity into the reinforcement learning setup, such as the need for inverse dynamics models and filtering. The practical implementation of these components could be challenging in some settings.\n\nQUESTIONS:\n1. Can the time reversal symmetry techniques be applied to a wider range of robotic tasks, or are they limited to tasks with explicitly reversible actions (e.g., peg insertion/removal)? It would be helpful if the authors could provide insight into how broadly these methods could be applied.\n\n2. Given that the paper does not include real-robot evaluations, how confident are the authors that their approach will perform well in a physical robot, particularly in environments with noise or variability in actuator performance? Future work could explore bridging this gap between simulation and real-world testing.\n\n3. In partially reversible tasks, how does the agent handle situations where only certain components of the state are reversible (e.g., door state vs. robot state)? A more detailed explanation of how partial reversibility impacts training would be useful.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}