Title: Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation
Venue: NeurIPS 2025
Accepted: yes
Keywords: tactile sensing

simulation

robotics
Abstract: Tactile sensing is crucial for achieving human-level robotic capabilities in manipulation tasks. As a promising solution, Vision-based Tactile Sensors (VBTSs) offer high spatial resolution and cost-effectiveness, but present unique challenges in robotics for their complex physical characteristics and visual signal processing requirements. The lack of efficient and accurate simulation tools for VBTSs has significantly limited the scale and scope of tactile robotics research. We present Taccel, a high-performance simulation platform that integrates Incremental Potential Contact (IPC) and Affine Body Dynamics (ABD) to model robots, tactile sensors, and objects with both accuracy and unprecedented speed, achieving a total of 915 FPS with 4096 parallel environments. Unlike previous simulators that operate at sub-real-time speeds with limited parallelization, Taccel provides precise physics simulation and realistic tactile signals while supporting flexible robot-sensor configurations through user-friendly APIs. Through extensive validation in object recognition, robotic grasping, and articulated object manipulation, we demonstrate precise simulation and successful sim-to-real transfer. These capabilities position Taccel as a powerful tool for scaling up tactile robotics research and development, potentially transforming how robots interact with and understand their physical environment.


Peer reviews:
comments: SUMMARY:
This paper presents a simulation framework for vision-based tactile sensors based on Incremental Potential Contact (IPC) and Affine Body Dynamics (ABD) in which IPC guarantees inversion- and intersection-free contact solutions and ABD is for efficient and precise simulation of stiff materials. This implementation is based on NVIDIA Warp and can achieve high FPS through parallelization. It supports a wide range of robotic tasks and environment setups and its performance is validated by many robotic tasks using tactile signals.

STRENGTHS AND WEAKNESSES:
Strengths: This work presented in this paper is very complete, including very detailed technical and implication details, sufficient experiment across many scenarios, and benchmarks compared with prior works. The proposed tactile simulation framework is more stable and can handle more complex physical interactions with smaller sim2real gap. It also achieves higher computational efficiency enabling more simulation environments running in parallel.
Weaknesses: This work is valuable but more like an incremental project. The combination of IPC and ABD has been implemented in other simulators for the same purpose and the tactile RGB image generation has also been explored previously. It is clear that this work outperforms the previous works but the technical novelty is marginal.

QUESTIONS:
The presentation is clear and I don't have any questions.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 1
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Taccel, a high-performance simulation platform tailored for vision-based tactile sensors (VBTSs) in robotics. VBTSs offer rich tactile feedback through gel pad deformations, but simulating their behavior remains challenging due to the complex mechanics of soft gel pads and the high computational cost. The proposed method addresses these challenges by integrating Incremental Potential Contact (IPC) and Affine Body Dynamics (ABD), achieving high-fidelity, collision-free simulations at >18 wall-clock speed.\n\nPowered by efficient implementation with NVIDIA Warp, Taccel supports large-scale parallelization. Its modular design also enables flexible sensor-robot configurations via user-friendly Python APIs, and it generates high-resolution RGB and depth-based tactile signals. The paper tests Taccel on diverse tasks, including object classification, grasping, and articulated manipulation, to demonstrate its strong capability for sim-to-real transfer, which outperforms prior simulators like TacMan in both precision and scalability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The paper proposes an IPC-ABD combined simulation to simulate vision-based tactile sensors (VBTS), which is novel, original, and significant. It improves the simulation accuracy compared to rigid body simulations like Tacto and includes the robot in the loop compared to FEM based simulator like DiffTactile. Authors have provided detailed mathematical derivations and justifications for when to use which simulation method.\n* The paper provides a thorough description of the simulation pipeline, covering all important aspects of VBTS, i.e. RGB images, normal maps, marker flow, and 3D point cloud information.\n* The experiment section provides comprehensive examples to demonstrate Taccel's capability of sim-to-real transfer and applications in robotic manipulation tasks through three main tasks: training object classification models, generating synthetic data with scalable parallelization, and manipulating articulated objects. They demonstrate the significance of VBTS and efficient learning with simulations like Taccel. \n* The paper and supplementary material are overall well-written and easy to follow. The video provides a good dissection and demonstrates the simulation scenes effectively. \n\n\n**Weakness**\n* The paper focuses on sensor and robot modeling, but doesn't elaborate on the object modeling. It mentions that Taccel can handle both rigid and soft objects, but could it handle object of different materials? would the simulation speed be affected? More testing examples and analysis would be helpful.\n* Clarification question: in \"object classification\" experiment, since the tactile data only provide local information, how does the model learn to differentiate different objects VS different parts of the same model? (Fig. 5 (a) shows nine mechanical parts, but the caption says ten. Is any of them considered as two objects?) Also, those parts seem to fabricated using the same material thus only differ in geometry. How would Taccel handle objects of different materials, e.g. wooden plate VS metal plate?\n* The paper mentions \"the simulation time scales linearly with the number of mesh vertices\". Does it mean simulating sensor contact with objects with more complicated geometries would take a longer time? How does it generalize if we want to simulate the gel pad with higher resolution, emulating a higher-resolution embedded camera? How does it generalize to daily objects? i.e. Taccel achieves 18x faster than wall clock time with \"bolt and nut\" task; how does it change with more complex but still common object manipulation tasks?\n\nQUESTIONS:\n* As discussed in the \"Strengths and Weaknesses\" section, can Taccel handle different object materials or objects with heterogeneous materials? If so, what is an efficient way to do that? If not, what are the challenges?\n* Part of the main sim-to-real challenges for VBTS come from (1) sensor degradation over time and (2) variance among different sensor configurations and among individual sensors of the same configuration. Would that be mitigated by using Taccel simulation? If so, how would Taccel efficiently simulate different sensor configurations and/or sensor degradation over time to reduce the sim-to-real gap?\n * Open-ended question for discussion: one weakness of VBTS is force estimation, as real sensors only use embedded cameras to capture the gel deformation. However, now given the simulation environment and the known full state of the sensor and objects, is it possible to derive force/torque measurement from the simulation, which then helps to train a NN to learn the mapping of tactile images to force estimation?  (Context: one of current methods for force estimation using VBTS is to collect a bunch of data pressing objects on the sensor with different forces, then train a NN. With Taccel, would it be easier to gather ground truth data with more variation, thus making the task easier?)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Taccel, a parallel simulation platform to simulate vision-based tactile sensors (VBTS) for robotics. Taccel combines Incremental Potential Contact (IPC) as a contact model, Affine Body Dynamics (ABD) for stiff materials, and Finite Element Method (FEM) for soft-body simulation of hyper-elastic soft gels used in VBTSs. Taccel enables accurate and high-speed simulation. The experiments demonstrate that Taccel can be used for applications such as generating synthetic data to train a model for object classification, simulating robotic grasping using multi-fingered dexterous hands with integrated VBTSs, and articulated object manipulation using tactile perception feedback.\n\nSTRENGTHS AND WEAKNESSES:\n[S1] The paper is generally well-written, and the proposed Taccel simulator and accompanying experiments are comprehensive. The authors motivate the use of IPC+ABD based on the limitations of prior work simulating VBTSs alternative simulation techniques. The performance gains of parallel simulation enabled by Taccel over single-environment simulators are impressive (16x). \n\n[W1] To the best of my knowledge, while the Taccel simulator is available on Github, most of the actual (IPC) simulation code is closed source (only compiled binaries are provided). Given that the authors appear committed to developing Taccel with the community (L187-189), I would encourage the authors to consider fully open sourcing Taccel. My main concern here is that IPC is notoriously slow, and understanding the optimizations made for IPC that achieves the performance reported in Taccel, would greatly benefit the community. Furthermore, it seems that Taccel uses SAPIEN-IPC? (an IPC implementation based on NVIDIA Warp that is fully open sourced.)\n\n[W2] It is difficult to directly compare the quality of Taccel’s simulation of VBTSs to existing VBTS simulators such as SAPIEN-IPC, Taxim, or DiffTactile, given the experimental results in Section 5 or 6. I think an experiment that compares directly pixel-wise tactile marker MSE between VBTS simulators would be helpful. While there is a quantitative result reporting SSIM between the simulated and real-world RGB tactile image, this robustness may be due to the depth -> RGB neural network, rather than quality improvements in tactile sensor simulation. \n\n- - -\n\nI am willing to raise my score if [W1] and [W2] are resolved.\n\nQUESTIONS:\nL61-63: I believe these numbers are for total effective FPS, instead of per-environment FPS? If this is the case, then I don’t think it’s accurate to describe 18x “real-time” performance in the abstract L10), because the single-environment FPS is still sub 1 FPS (Figure 4). \n\nL309: What are execution-recovery switch counts? This metric is not explained in the subsection.\n\nFigure 2(a): It looks like only two rigid planes are interacting with the bolt (while the Franka gripper is only used for visualization). Does the task fail if the finger collision shapes are smaller (similar to what is used in PyBullet and Sapien)? What collision simplifications are made by Taccel, such as for the articulated rigid body tasks in Figure 7? \n\nSAPIEN-IPC is also implemented using NVIDIA Warp. Is Taccel based on SAPIEN-IPC (which appears fully open sourced)? What are the major differences between Taccel’s implementation of IPC and SAPIEN-IPC that enable faster simulation (Figure 2 (d))? L251 mentions that SAPIEN-IPC’s use of FP32 leads to convergence issues, while Taccel uses FP64 (Figure 4). I would expect use of FP64 to lead to slower simulation (and larger memory requirements). Does FP64 enable fewer solver iterations or larger timesteps?\n\n---\n[minor]\n- References should not in the abstract?\n- L37-38: “contactsd” spelling?\n- L60: “warp” is lowercase?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Taccel, a high-performance simulation platform for vision-based tactile sensors (VBTSs) integrated with robotic systems. The key technical contribution is the combination of Incremental Potential Contact (IPC) and Affine Body Dynamics (ABD) to achieve both physical accuracy and computational efficiency. The platform achieves an 18-fold speedup over real-time simulation across thousands of parallel environments on a single GPU. The authors validate their system through three applications: object classification with sim-to-real transfer, large-scale robotic grasping dataset generation, and articulated object manipulation. The paper demonstrates that Taccel can generate realistic tactile signals while maintaining stable, penetration-free physics simulation at scale.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n+ The unified IPC-ABD formulation seems mathematically sound and addresses key limitations of existing simulators. The use of IPC guarantees intersection-free trajectories while ABD efficiently handles rigid components, avoiding the computational overhead of treating everything as soft bodies.\n+ The ability to simulate 4096+ parallel environments with tactile sensors is a significant achievement. The performance benchmarks showing 915 FPS for low-resolution tactile simulation represent a major improvement over existing solutions like SAPIEN-IPC.\n+ The paper provides thorough validation across multiple dimensions - physics accuracy (bolt-nut assembly, soft block pressing), tactile signal fidelity (SSIM of 0.93), and real-world task performance (Tac-Man manipulation with 1.1% error compared to real execution).\n+ The sim-to-real transfer results (70.94% accuracy on real mechanical parts after training only on synthetic data) demonstrate the practical value of the simulator for developing tactile perception systems.\n+ The provision of Python APIs and support for standard formats (URDF) makes the tool accessible to the robotics community.\n\nWeaknesses:\n- Limited tactile sensor models: While the paper focuses on GelSight-type sensors, it's unclear how easily the framework extends to other VBTS designs (e.g., different gel materials, alternative optical configurations). The RGB signal generation relies on a DNN trained on only 200 real images, which may limit generalization.\n- Despite the impressive speedup, the system still requires high-end GPUs (H100) for optimal performance. The scalability analysis doesn't adequately address performance on more accessible hardware or the memory-computation tradeoffs.\nIncomplete comparisons: The paper lacks detailed comparisons with some recent simulators. Most notably, there's no comparison with TacEx (Nguyen et al., 2024, TacEx: GelSight Tactile Simulation in Isaac Sim -- Combining Soft-Body and Visuotactile Simulators), which also combines soft-body and visuotactile simulation in Isaac Sim. This is a significant omission given the similar goals.\n-The paper doesn't thoroughly discuss when and why the simulation might fail to match real-world behavior. The sim-to-real gap in object classification (86.5% -> 70.94%) suggests room for improvement.\n- The validation focuses primarily on static/quasi-static scenarios. Dynamic manipulation tasks with rapid contact changes or multi-contact scenarios are underexplored.\n\nQUESTIONS:\n* How does Taccel compare to the recently presented TacEx system (Nguyen et al., CoRL 2024 Workshop), which also combines soft-body simulation with visuotactile rendering in Isaac Sim? Could you provide a detailed comparison in terms of accuracy, performance, and features?\n* How difficult would it be to extend Taccel to support other VBTS designs beyond GelSight-type sensors? What modifications would be required for sensors with different optical properties or gel geometries?\n* The current validation focuses on relatively slow, controlled movements. How does the simulation accuracy degrade under dynamic conditions with rapid contact formation/breaking or high-velocity impacts?\n* What is the relationship between memory usage and the number of parallel environments? Is there a practical upper limit beyond GPU memory constraints?\n* How sensitive are the results to the choice of material parameters (Young's modulus, Poisson's ratio)? Do you provide guidance for parameter identification from real sensor data?\n* Are there plans to support multi-GPU execution for even larger-scale simulations? What would be the main challenges in implementing this?\n* For online robot learning applications, what is the minimum hardware configuration needed to achieve real-time performance with a reasonable number of environments (e.g., 100-1000)?\n\nLIMITATIONS:\nThe authors acknowledge computational demands as a limitation, identifying PCG-based linear system solving as a bottleneck. However, several other limitations deserve mention:\n\n- The fidelity of tactile signal generation depends on the quality of the trained DNN model, which may not generalize to all surface textures or contact conditions\n\n- The Neo-Hookean material model may not accurately capture all aspects of real gel pad behavior, particularly under large deformations\n\n- The current implementation appears limited to single-GPU execution, which may constrain very large-scale simulations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}