Title: Enhancing Tactile-based Reinforcement Learning for Robotic Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning

tactile sensing

representation learning

robotic manipulation
Abstract: Achieving safe, reliable real-world robotic manipulation requires agents to evolve beyond vision and incorporate tactile sensing to overcome sensory deficits and reliance on idealised state information. Despite its potential, the efficacy of tactile sensing in reinforcement learning (RL) remains inconsistent. We address this by developing self-supervised learning (SSL) methodologies to more effectively harness tactile observations, focusing on a scalable setup of proprioception and sparse binary contacts. We empirically demonstrate that sparse binary tactile signals are critical for dexterity, particularly for interactions that proprioceptive control errors do not register, such as decoupled robot-object motions. Our agents achieve superhuman dexterity in complex contact tasks (ball bouncing and Baoding ball rotation). Furthermore, we find that decoupling the SSL memory from the on-policy memory can improve performance. We release the Robot Tactile Olympiad ($\texttt{RoTO}$) benchmark to standardise and promote future research in tactile-based manipulation. Project page: https://elle-miller.github.io/tactile_rl.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a new method for in-hand robot manipulation tasks, which introduces two dynamics objectives: tailored reconstruction and multi-step forward dynamics with auxiliary memory to address the issue of 1) the policy prematurely converging to features, 2) the difficulty of learning from sharp transitions and sparse nature data. The experiments show that the new methods outperform the previous work in multiple in-hand manipulation tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. Tailored reconstruction and multi-step forward dynamics incorporate extracted learning objectives for the encoder, enabling the model to effectively capture and extract features from the observations, thereby facilitating the performance of downstream tasks.\n2. The incorporation of the novel loss does not significantly enhance the complexity of the pipeline. These straightforward objectives maintain clarity and simplicity within the pipeline.\n\n## Weakness\n\n1. The experiments show that the overall performance increased, and the ablations show that the new objectives benefit the performance. But the proposed solutions for the hypotheses are not validated in the work. \n    1. Tactile reconstruction softens the issue that learning \"prematurely converges to features\" is not discussed, as figures of value/policy loss show the gradients converging more smoothly.\n    2. Also, no experiments show that \"Multi-step forward dynamics\" helps the learning from \"sharp transitions and sparse nature\n\n2. Figure 1 is very confusing, making it hard to trace the details of the main contribution.A more detailed caption should be added to the paper.  I am assuming\n    1. the blocks/squares in the right two subfigures (ssl) from left to right are the feed-forward directions (even though they do not have arrows)\n    2. SSL: tactile reconstruction: the upper sequence is a feed-forward network, and the lower arrow is the ground truth. L_BCE is computed by MSE from the upper and the lower\n    3. SSL ulti-step forward dynamics: similar confusion to the above.\n\nQUESTIONS:\n1. Are the two objectives mainly to help the training of the encoder? The decoder (d),  the forward model (f), and the predictor (p) are not used during inference ?\n2. Are the two objectives designed for the encoder only? Not the value network and the policy?  If so, how can we prevent these two networks from falling into the problem in the hypotheses?\n3. How does the tactile reconstruction prevent the model from \"prematurely converging to features\"? How does \"Multi-step forward dynamics\" help learning from sharp transitions and the sparse nature of tactile data more easily? For me, this seems to be hard to prove, but it is claimed in the paper that the new methods address the problems somehow. It is important to prove it.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on enhancing tactile-based (RL) for robotic control by addressing the challenge of effectively exploiting sparse tactile sensory signals. The paper proposes training of RL policies with a binary reconstruction of tactile data streams via an encoder architecture, hypothesizing the higher likelihood of signal value and sim2real transfer over continuous force readings. They further propose introduction of auxiliary reconstruction and prediction objectives and separate memory for the RL task policy and auxiliary objective policy in order to encourage the policies not to ignore the haptic signals in favor of richer proprioceptive signals. To test the hypotheses and validate their claims, the robot tactile olympiad simulation benchmark is introduced and results demonstrate the super-human performance on bounce and baoding tasks in simulation.\n\nSTRENGTHS AND WEAKNESSES:\nOverall interesting hypothesis, well conducted research, and strong results/discussion. The auxiliary objectives, novel architectures, and new benchmark appear to be positive contributions to the field and I appreciate the focus on scientific discussion and suggestions to others working on similar problems. Of course it would be great to see sim2real results and any result obtained in a single impulse-based simulation on dynamic interactive tasks must be taken with a grain of salt. That said, I see clear impact and novelty here and support acceptance of the work.\n\nPros:\n- novel formulation \n- auxiliary objectives to force the policy to encode the haptic data\n  - future state prediction is a good objective to encourage learning from sparse signals\n- separate memory for auxiliary haptic objectives allow for optimization on a wider distribution of data offline\n- significant improvements in capability on test tasks\n  - simulated agent is superhuman in bounce and baoding\n- robot tactile olympiad dataset\n- binary tactile data does appear to be more transferable between sim and real in the short term\n- Good hyper-paramter tuning to ensure validity of results\n- Nice focus on and interesting format of discussion section adds scientific value to the work\n  - I like the \"translation to practical recommendations\" as a focus on advancing the field\n\nCons:\n- binary contact signals are limiting for more practical tasks\n- no real world transfer experiments\n  - because physics simulation is impulse based approximation, it isn't clear if superhuman performance would be possible in the real world vs. simulation\n- large variances for complex task policies\n- No unified hyper-paramter setting for all tasks indicates challenges in generalization\n\nQUESTIONS:\nI am interested in the authors' opinions on the value of pursuing and optimizing for superhuman results in simulation. I think it is great to see such results and the videos help to put the quantitative analysis into context, but I know first-hand how simulation inaccuracies can enable non-physical behaviors which policies can and will latch onto. For example, oscillatory impulses applied by contact resolution between two frames at high frequency could masquerade as \"bouncing\" or \"vibration\". Concretely, imagine how different soft contacts between a human hand and ball would be from a rigid approximation in an impulse based engine. The question boils down to: \"how far should we trust impulse-based simulation models with regards to highly dynamic tasks?\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper posits that the current tactile manipulation policies entangle the proprocioptive and tactile representations making it unclear and also hard to attribute the benefit of \"tactile\" in tactile manipulation. To investigate this, the authors propose to disentangle and learn SSL methods on top of the tactile states as a form of auxiliary loss to guide the RL policy learning. Specifically, they consider tactile dynamics and tactile reconstruction as 2 SSL tasks.\nFurther, to evaluate the method, the authors propose a benchmark - RoTO: Robot Tactile Olympiad that includes 3 tasks, namely Find, Bounce, and Baoding involving parallel gripper (franka arm) and dextrous manipulation (shadow arm).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Well-written paper\n\n- The idea of specifically learning tactile dynamics is an interesting one and a good contribution to the field of tactile manipulation.\n\n- Thorough hyperparameter search including learning rate(!), which is often missing in a lot of RL papers.\n\n\n**Weaknesses**\n\n\n1. Related works miss several tactile-based dynamics models such as RoboPack [1] and Swing Bot[2]. Citing & discussing them would be helpful for the reader.\n\n2. Lack of Sim2Real experiments. This is acknowledged by the authors, however, I must note that the correlation between simulation experiments working versus the policy actually working on the real robot is quite large, especially for dexterous manipulation, hence I do consider this limitation as a major drawback of the work.\n\n---\n\n**References**\n\n[1] RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing, Bo Ai et al., RSS 2020.\n\n[2]SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation, Chen Wang et al., IROS 2020\n\nQUESTIONS:\n3. Figure 1 (right) -- the architecture for multi-step forward dynamics is confusing since there are both L1 and L2 losses being taken. I presume that this was a typo and it's L2 loss everywhere (based on the multi-step forward dynamics subsection). But I wanted to confirm if my assumption is correct.\n\n4. For the forward dynamics, is there any prioritized sampling performed to reduce the imbalance present in the tactile state transitions?\n\n5. Can the authors show the plots for the ablation study on $n_f$ for at least one task? I'm curious how the values were chosen. Was the last $n_f$ value that didn't result in a significant increase from $n_f-1$ selected as the optimal value?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a general-purpose self-supervised learning methodology that effectively leverages tactile input for downstream robotic control tasks in RL. Firstly, the manuscript provides a novel application and analysis of tailored reconstruction and multi-step dynamics objectives that enable the agent to leverage its tactile observations. Secondly, the proposed method proposes to separate the auxiliary memory from the RL rollout memory to stabilise and strength the self-supervision updates. Experimental results show the superiority of the proposed self-supervised agents compared to RL-only agents on three kinds of tasks: Find, Bounce, Baoding in Issac Lab.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The proposed method show the superiority over RL-only agents on multiple kinds of tactile-based complex manipulation tasks. \n\n2. The authors provide various aspects of discussion to analyze the advantage of the proposed dynamics objective and self-supervised learning strategy.\n\n3. Supplementary materials are provided with detailed implementation details and additional experimental results.\n\n### Weaknesses\n1. A major concern is the organization and presentation of the manuscript, which largely impact the readability. For example, the Method section contains a lot of symbols without providing a notation summary table to explain the meaning of each variable. Moreover, the captions of figures do not explain the figure (e.g., the pipeline of method and the meaning of the symbols in Figure 1) in detail. Thirdly, what is the total training objective and how the self-supervised learning cooperate with the RL objective are unclear from Figure 1. Fourthly, In the experimental results, the authors only provide the comparison results in the form of figures instead of tables, making the comparison indirect. I strongly suggest the authors to refine and reorganize their manuscript to improve the clarification and readability.\n\n2. The design of the proposed method seems to lack enough technical details and contents. The core contribution is the self-supervised training strategy, which has been widely adopted in previous works. The separate auxiliary memory is also an important contribution as described in Line 77-80. However, the authors only introduce it briefly in Section 3.3. \n\n3. In Line 70, the authors claim that the proposed method is supposed to have low sim2real gap, however, the manuscript does not contain the real-world experiments, as indicated in the Limitation section. These makes the adaptability of the proposed method to the real world doubtful.\n\nQUESTIONS:\nPlease see Weaknesses section. In my opinion, the poor organization of the manuscript, the lack of real-world experiments, and the insufficient method design are the major issues of the manuscript. Therefore, I lean to recommend rejection in the current version of submission.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}