Capturing forceful interaction with deformable objects using a deep learning-powered stretchable tactile array

Domain: earth_science
Published: 2024-11-04
Editorial outcome: minor_revision

Abstract
Capturing forceful human manipulation during manipulation benefits applications like virtual reality (VR), telemedicine, and robotics. Replicating full hand-object states with complete geometry is challenging because of the occluded object deformations. Here, we report a visual-tactile recording and tracking system for manipulation featuring a stretchable tactile glove with 1152 force-sensing channels and a visual-tactile joint learning framework to estimate dynamic hand-object states during manipulation. To overcome the strain interference caused by contact with deformable objects, an active suppression method based on symmetric response detection and adaptive calibration is proposed and achieves 97.6% accuracy in force measurement, contributing to an improvement of 45.3%. The learning framework processes the visual-tactile sequence and reconstructs hand-object states. We experiment on 24 objects from 6 categories including both deformable and rigid ones with an average reconstruction error of 1.8 cm for all sequences, demonstrating a universal ability to replicate human knowledge in manipulating objects with varying degrees of deformability.

Introduction
Human-machine interaction (HMI) systems serve as gateways to the metaverse, acting as bridges between the physical world and the digital realm. A natural user interface in HMI allows humans to perform natural and intuitive control1. Although the non-forceful interfaces such as hand gestures (Fig. 1A(i)) can be tracked using technologies like inertial measurement units (IMU)2, electromyography (EMG) sensors3,4, strain sensors5,6, video recording7 and triboelectric sensors8, the forceful interfaces such as interaction with objects, i.e., the human manipulation, are less explored9,10. Capturing forceful human manipulation has extensive potential applications, such as virtual reality (VR)11,12, telemedicine13, robotics14,15, and contributes to real-world understanding for large artificial intelligence (AI) models16. Replicating the hand-object interplay is the first step to applying human manipulation knowledge in these applications. However, the hand-object states captured in previous research were far from complete. They mainly explore tasks like semantic recognition and spatial localization to predict object category and position (Fig. 1A(ii))17–20. Imagine a general manipulation case: when a human rubs plasticine to a desired shape, he needs to sense the surface deformation and track the object’s geometric states. Tactile perception takes precedence in analyzing deformations within the contact area21, while visual perception is utilized to estimate the overall object states (Fig. 1B(i))22. Thus, to capture the comprehensive information during manipulation in a human-like way23, a system that can record the visual-tactile sensory data and estimate the fine-grained hand-object states is desired.

Recording tactile data in hand-object interplay is challenging, particularly when recording forces on stretchable interfaces during interactions with deformable objects. On the one hand, the tactile array should feature a high-density distribution of force-sensing units to cover multiple contact areas during object operation. A natural choice is to integrate a tactile array into a wearable glove using textile techniques24–26. On the other hand, the tactile array should be a stretchable interface, ensuring conformal contact with deformable objects27. However, when in contact with a deformable surface, the extension or bending of the stretchable tactile array could encounter undesired interference with output signals due to the increasing strain (Fig. 1B(ii))28–32. Unlike traditional rigid-rigid or flexible-rigid interfaces without stretchability, it is suggested that the normal force cannot be independently measured at the stretchable surfaces, otherwise the strain will significantly impair precise force measurement33. To achieve strain insensitivity, previous studies have typically solved the problem from structural or material perspectives. Structurally, methods include stretchable geometric structures34–37, stress isolation structures38–41, and Negative Poisson’s ratio structures42–44. Material strategies include strain redundancy techniques45–48, localized microcracking techniques49–51, and nanofiber network encapsulation technique52–54. These techniques belong to the source-protection approach, concentrating on the “input end” of the sensor and aiming to reduce or eliminate the influence of strain interference on tactile arrays in an open-loop non-adaptive manner55–57. However, these techniques have three limitations: no quantitative assessment of the strain received, no quantitative assessment of the strain suppressed and no measurement standard considering real-time testing conditions. Thus, a closed-loop adaptive tactile data recording approach is desired, where the closed-loop monitoring quantitatively detects and suppresses the strain interference and adaptive force estimation highly suits the deformable interface with unpredictable or high degrees of freedom.

Although closed-loop adaptive perception of tactile data facilitates the estimation of object deformation at the contact area, it is insufficient for recovering the complete object state, as some parts of the object remain out of contact. Thus, it is naturally necessary to adopt the assistance of global signals, such as visual perception, to obtain the full object geometry58. Previous works on visual-tactile joint learning have primarily relied on camera-based tactile sensors59–61, benefiting from their locally high-resolution and 2D grid-like data format. Along with visual images, they have been utilized for hand pose estimation, in-manipulation object pose estimation, and geometric reconstruction62,63. However, these visual-tactile models capture the hand-object interaction in static settings and do not consider the temporal consistency of hand movements and object deformation64. Supplementary Table 1 compares visual-only, tactile-only, and visual-tactile modalities for object understanding and manipulation.

To capture the forceful human manipulation of deformable objects, this paper proposed a visual-tactile recording and tracking system for manipulation named ViTaM, which employs a high-density, glove-shaped stretchable tactile array for force recording and a deep learning framework for visual-tactile data processing and hand-object state estimation. Especially, the stretchable tactile array works in an output-focus sensing paradigm by measuring forces under different strains in a closed-loop adaptive manner. Based on the proposed negative/positive stretching-resistive effect, quantitative symmetric response detection and suppression evaluation of strain interference are achieved, enabling accurate force measurement on the stretchable interface with an accuracy of 97.6%, which has an improvement of 45.3% compared with the uncalibrated measurements. Meanwhile, a point cloud sequence as visual observations capture the entire interaction process. During data processing and hand-object state estimation, the learning framework adopts two distinct neural network branches to encode visual and tactile information respectively, and reconstructs the fine-grained surface deformation and the complete object geometry. To demonstrate the generalization ability of the learning framework, we select 24 objects from 6 categories, including both deformable and rigid ones, and we can achieve an average reconstruction error of 1.8 cm over all sequences. This work marks a revolutionary advancement of perception tools for human manipulation, takes a step towards a more generic recording approach for both rigid and deformable objects, completes the last mile of forceful interaction by machine intelligence, and improves the learning framework for linking the physical world and the digital realm.

Results
Overview of the ViTaM system
The design of the ViTaM system is rooted in the idea of capturing fine-grained information during forceful interaction with deformable objects. It records the manipulation process with a proposed high-density, stretchable tactile glove, and a 3D camera, and estimates the hand-object state at the geometric level with a proposed visual-tactile joint learning framework. When the user interacts with objects, the hand-object state at the contact area is recorded by the tactile glove. It has high-density tactile sensing units with a maximum of 1152 channels distributed all over the palm and can accurately capture the force dynamics with a frame rate of 13 Hz on the stretchable interface between the hand and object during interaction (Fig. 1B(iii)). Meanwhile, the hand-object state at the non-contact area is recorded by a high-precision depth camera. The captured force measurement and point cloud sequence are processed by the visual-tactile learning model proposed in this article, facilitating cross-modal data feature fusion, ultimately enabling the tracking and geometric 3D reconstruction of manipulated objects with varying deformability, including both deformable (e.g., elastic and plastic) and rigid ones (Fig. 1B(iv)).

Design and fabrication of the tactile glove
The tactile glove contains the following modules (Fig. 2A): tactile sensing blocks, a fabric glove, flexible printed circuits (FPCs), a multi-channel scanning circuit, a processing circuit, and a bracelet. The naming labels of tactile sensing blocks are given in Fig. S1. Three types of FPCs connect the finger and palm sensing areas with the multi-channel scanning circuit and processing circuit (Fig. S2). Fixing and encapsulation methods are detailed in Fig. S3. The modular design allows for optimal performance, on-demand density expansion, and detachability. The multi-channel scanning circuit (Fig. S4), which contains a force sensing circuit (Fig. S5A) and strain interference detection circuits (Fig. S5B), supports up to 1152 sensing units per frame, with the prototype demonstrating 456 sensing units. Additionally, a custom data transmission protocol ensures efficient and adaptable data transfer (Fig. S6). In Fig. S7A, the tactile glove demonstrates excellent wearability and conformability through two distinct gestures. To validate the yield rate of the prototype, external forces were applied to each area of the five fingers (Fig. S7B) and palm (Fig. S7C) of the tactile glove. Upon calculation, the yield rate was determined to be 97.15%, which is sufficient to meet the requirements of most human-machine interaction applications. Besides, the estimated costs of the tactile glove and hardware are $3.38 (Supplementary Table 2) and $26.63 (Supplementary Table 3), respectively, fostering widespread public acceptance16. In the future, due to the simplicity of the processing procedure and automation advancements in processing equipment (such as sewing machines), there is significant potential for mass production of this tactile glove.

The tactile array consists of multiple tactile sensing blocks, and each block includes a positive strain sensor, a negative strain sensor, and a force sensor array (Fig. 2B(i)). The negative and positive strain electrodes are connected with the assembled composite film, respectively (Fig. 2B(ii)). Besides, the conductive fabric wires are sewn onto the assembled composite film, forming the row electrode array and column electrode array of the tactile force sensor array, with the rows and columns positioned perpendicular to each other. The fully woven wiring method is given in Fig. S8A and the overlap of a row electrode, an assembled composite film, and a column electrode forms a tactile sensing unit (Fig. S8B). As shown in the cross-sectional view of Fig. 2B(iii), the electrodes are tightly shuttled between the films, avoiding the use of an adhesive layer and showing better assembling. Different from conventional techniques like photolithography or screen print, which assemble the top and bottom electrode layers on the two sides of the sensing film, this fully woven wiring method requires no adhesive for layer contact, so interlayer delamination will not occur, leading to better reliability, conformality, and wear resistance. Adjacent blocks share the row and column electrodes (Fig. S9).

The above assembled composite film is composed of stacked positive effect membrane (the orange layer) and negative effect membrane (the blue layer), each linked to a strain detection module via electrode pairs. Figure S10 illustrates the fabrication of negative and positive effect membranes. The negative stretching-resistive effect was first proposed in our previous work65. The positive or negative effect of a film is determined by the content of carbon nanotubes (CNTs). A higher weight ratio of CNTs than 3.3 wt% in the natural latex substrate leads to a negative stretching-resistive effect, while a lower CNT weight ratio results in a positive effect. In this experiment, the CNT contents of 5 wt% and 2.9 wt% are chosen because they show comparable resistance changes with similar amplitudes but opposite changing trends when the strain increases gradually from 0 to 50% (Fig. 2C). This phenomenon enables the measurement of the strain variation across a wider range, covering the maximum extension region of the fingertip of 40%66.

Adaptive strain interference suppression method
To improve the accuracy of force measurements when manipulating deformable objects, an adaptive strain interference suppression method is proposed for the detection and suppression of strain interference on the stretchable interface (Fig. 2D). Generally, the tactile array outputs a signal variation with the applied force changing, and a curve depicting the relationship of the forces and the outputs is obtained (called force estimation curve). Inferring a force based on the specified and pre-tested force estimation curve is the conventional open-loop measurement method for tactile sensors. However, in this work, in addition to the forces exerted on the composite film, the strain interference is also detected in a closed-loop manner. Prior to force calibration, several tests are performed to measure the force estimation curves under several different strains (shown in the next section). The following steps shown in Fig. S11A offer the detailed process: (i) First, how can the presence of strain interference be determined? The positive and negative effect membranes are connected with the strain detection module, respectively, constituting a novel dual input mode. As soon as an increased strain interference occurs, the output voltage of the strain detection circuit promptly shows a synchronized rise edge and further generates a warning flag to alert the existence of strain interference; (ii) Second, how can the magnitude of the strain interference be quantitatively determined? If strain interference exists, the strain εx can be inferred from the relative resistive variation curve based on Fig. 2C. This stage determines the appropriate calibration coefficient in the subsequent calibration step; (iii) Third, how to obtain the force estimation curve under εx? Since it is impractical to enumerate force estimation curves for all possible strain interferences, this study proposes a method called “local domain curve interpolation” to update the force estimation curve corresponding to εx. As shown in Fig. S11, identifying the two strain values closest to εx among the known conditions (named ε_up and ε_bottom) and their corresponding force estimation curves (named curve_up and curve_bottom). Then, a new curve is interpolated between curve_up and curve_bottom in proportion to the distance between εx and ε_up, ε_bottom. This interpolated curve (named curve_x) represents the force estimation curve corresponding to the strain interference εx; (iv) Fourthly, using curve_x to calculate the force under the strain interference εx. The examples in Fig. S11B illustrate the obtained interpolated force estimation curve when the ε_up is 10%, ε_bottom is 0%, and εx is 2.5%, 5%, and 7.5%, respectively.

In this adaptive system, the force measurement improves from a single-factor derivation of voltage alone to a dual dependent variable control of voltage and strain, surpassing the traditional single-electrode approaches and improving the accuracy of force estimation. In previous studies67,68, the resistance rise or fall of the piezoresistive sensor cannot determine whether the sensor is subjected to strain, load force, or both. As for our method, the two inputs that are simultaneously monitored show opposite change trends, preventing false judgments and offering sensitive responses even with the slightest strain disturbances. Supplementary Table 4 compares the current methods for strain interference suppression and our method, revealing the proposed way is more straightforward without additional strain redistributing/releasing microstructures or stress-absorbing materials. Therefore, this strategy has the potential to expand from prototypes to applications in wider fields that contain strain interference.

Characterization of the optimized tactile sensing performance
The assembled composite films shown in Fig. 3A present good deformation adaptiveness when it is rolled up and rounded up, respectively. Figure 3B is the scanning electron microscope (ULTRA55, Zeiss, Oberkochen, Germany) image of the membrane cross-section, displaying the beneficial inhomogeneous microstructure for the high sensitivity to force responses. The image of the transmission electron microscope (TEM) clearly shows the alignment of CNTs in the substrate of natural latex occurs along the direction of strain generation (Fig. 3C).

To evaluate the strain interference suppression method, a tactile sensing block is attached to the knuckle inside the glove (Fig. 3D). As the finger bends backward and recovers, a pair of relative resistance variations with opposite changing trends but the same amplitudes bring about rise edges when the strain raises (Fig. 3E). It is also simulated by the Multisim software and the results are displayed in Fig. 3F. To construct the calibration module of the strain interference suppression method, the relationship between the output voltage of the tactile array and the applied force is tested under different strains. Figure 3G shows the strain from 0 to 30% sharply deteriorates the output voltages. Especially when the strain exceeds 10%, the voltage deviates significantly from the measured value under no strain, so it urgently requires dynamical calibration under the current strain. Due to the impracticality of enumerating all output voltage curves under various strain interferences, the calibration module fits a curve between the two closest curves, which serves as the updated one to output the calibrated estimated value of the force (Fig. 3H). It is worth mentioning that the hardware circuit with high-precision ADC chips of 12-bit can distinguish a subtle output increase even under large strain interference. The relative error between the estimated forces and the applied forces is depicted in Fig. 3I, where the estimated forces without calibration are generally smaller than the applied forces, and the deviations increases severely with increasing strain. After calibration, the estimated forces establish a strong consistency with the applied forces, proving the effectiveness of the proposed strain interference suppression strategy with an accuracy of 97.6% and an improvement of 45.3% compared with the uncalibrated measurements.

The mechanical responses of the tactile array under a 5% strain are discussed. Figure 3J shows the tactile array has a high sensitivity of 271.26 kPa−1 within the range of 100 kPa. It can meet the general range of hand grasping pressure below 20 kPa, which is equivalent to a palm carrying a weight of 20 kg. As illustrated in the inset image of Fig. 3J, the tactile array displays a significant relative resistive variation even when the pressure is below 1 kPa. This level of pressure is comparable to lifting a strawberry by the fingertip. The relative resistance variations to continuous but varying pressures are shown in Fig. S12. The response time and recovery time are measured as 52 ms and 80 ms, respectively (Fig. S13A). The relative resistance variation during the pressure loading and releasing is given in Fig. S13B. Besides, the signal-to-noise ratio of the tactile array is calculated to be 70.64, and the minimum sensing limitation is 36 Pa, which means the array can detect subtle changes when manipulating light objects, such as detecting a small weight of 50 mg with a contact surface of 3.5 mm × 4 mm. Moreover, Fig. 3K demonstrates the repeatability of the tactile array under a load pressure of 5 kPa over 2000 times, indicating an extended lifespan in practical applications. The negligible drifting after extensive use with human hands was discussed in Fig. S14. Besides, each tactile sensing block was calibrated point by point by the calibration platform shown in Fig. S15. To improve the system-level output response of the tactile glove, the influence of different Rg on the output voltage was investigated (Fig. S16A). When the Rg is set to 0.1 times the resistance to be measured, the percentage of resistance decrease β shows the maximum tolerance. Similarly, the reference voltage Vref of 2.5 V can offer the maximum voltage variation range △V, as shown in Fig. S16B. The final output curve of the tactile glove is given in Fig. 3L and the consistency from different tactile sensing areas has been well demonstrated (Fig. S17). The comparison of the current tactile gloves and our work is given in Supplementary Table 5.

Analysis of the interactions between the tactile glove and deformable objects
To validate the tactile glove, a dynamic dumpling-making task is performed using soft plasticine as a highly deformable object. The task involves kneading the plasticine into a ball, pressing it into a flat shape (acting as the dumpling wrapper), and pinching the wrapper together. Firstly, as the plasticine ball is rolled by the palm, the VR interface displays the distribution and values of the detected forces (Fig. S18 and Supplementary Movie 1). Figure 4A shows the normalized pressure of the blocks on the palm sensing area (called palm blocks) during kneading. Figure S19A shows the Spearman correlation analysis result of the palm blocks, where the correlation coefficients higher than 85% are highlighted (indicating a strong correlation). Figure S20A shows the blocks with strong correlations in this step. Secondly, the palm applies a substantial force to the plasticine ball by pressing (Fig. 4B) and the normalized pressure with strain interference correction is higher than the uncorrected one. Figure S19B and S20B show that more palm blocks contribute to the pressing process; thirdly, the operator folds the wrapper in half and then tightly pinches the edges together with the thumb and index finger (Fig. 4C). The normalized pressure of pinching shows the corrected curve displays an obvious increase in the three sub-stages, which could be caused by the significant strain and the reduction in the uncalibrated compressive force. The result of Spearman correlation analysis between the blocks on the finger sensing areas (called finger blocks) and the palm blocks is given in Fig. S19C, showing that block 1-1 and block 2-1 are highly collaborative with a coefficient of 92.3%, which is consistent with the pinching operation (Fig. S20C). Additionally, correlations between all blocks were analyzed as the operator grabbed a piece of foam board and a toy, and rolled the plasticine (Fig. S21).

Moreover, the tactile sensing blocks before and after strain interference calibration were also explored in operations requiring the cooperation of fingers and the palm. For example, a sponge is repeatedly pinched and released (Fig. 4D, Supplementary Movie 2). This operation without correction only involves six active finger blocks and nine active palm blocks with correlation coefficients greater than 85% (Fig. 4E(i)). Calibration revealed two more active finger blocks and five more active palm blocks, as depicted in Fig. 4E(ii). Figure 4F(i) illustrates normalized pressure changes of the active blocks before calibration, and Fig. 4F(ii) reveals blocks with minor pressure changes after strain interference correction. Spearman correlation results without and with calibration are shown in Fig. 4G(i) and Fig. 4G(ii), respectively. Block 3-1, located on the distal phalange of the middle finger, exhibited the strongest correlation coefficients with other blocks. After calibration, additional correlations emerge, indicating the involvement of all finger blocks in sponge grasping—particularly blocks 2-2, block 5-1, block 5-2, and block 5-3. Some blocks, like block 3-1 and block 2-1, exhibit increasing correlation coefficients than 85% after calibration, underscoring heightened synergistic effects between involved blocks. The increased number of strong correlations in Fig. 4H demonstrates how calibration enhances the exploration of dependencies between different fingers and the palm, even if strain interference exists. Furthermore, the tactile glove facilitates shape estimation during manipulation, evident in grasping various objects—both soft (a plastic dropper, a towel, a plastic bottle) and hard (a paintbrush, a spoon, a small needle)—with discernible force responses along the object edges in the VR interface (Fig. S22).

The interference of hand pose is also considered. Figures S23–S25 respectively compare the normalized pressure curves of three typical actions—kneading dough, grasping a sponge, and pinching a paper cup—between empty hand poses and interactions with real objects. The normalized pressure curves during real interactions are 12, 16, and 6 times those of the empty hand poses. These noises with lower amplitudes can be easily filtered out by the proposed visual-tactile jointly learning framework, which possesses denoising capabilities. Under a supervised learning setting, signals related to the supervised task (e.g., contact reconstruction) are enhanced, while unrelated signals are attenuated.

Visual-tactile learning of human manipulation
With the tactile glove, we are interested in uncovering the dynamics of the hand-object state, particularly deformable objects susceptible to strain interference during manipulation. Estimating deformed geometry is inherently challenging due to near-infinite degrees of freedom in the deformable region. The glove can measure the distributed forces that cause the deformation at the contact region, but it only covers a partial object surface, though high-density and distributed. Thus, we also need visual observations to help recover the complete object geometry. Such visual-tactile mechanisms in manipulation are similar to the human cognitive process23.

We introduce the visual-tactile learning framework during manipulation for hand-object reconstruction and tracking, adept at reconstructing complete object geometry even amid highly non-rigid deformation. Figure 5 provides an overview of our model. To assess the learning framework, a visual-tactile dataset is curated, comprising 7680 samples involving 24 objects across 6 categories. Among them, sponges, plasticine, bottles, and cups are deformable objects, while folding racks and scissors represent rigid objects. Each object undergoes 20 touches and 16 camera views. Training data are generated from RFUniverse69, a finite element method (FEM)-based simulation environment, while the test set is collected from the real world. We present the in-contact object reconstruction for two elastic objects (sponges) and a rigid object (a scissor). The quantitative results are reported in Supplementary Table 6, and the qualitative results are shown in Fig. 6A. In Supplementary Table 6, we can see that the performances of our model on both simulation and real data are remarkable since the calculated chamfer distances are within an acceptable range. We can see in Fig. 6A that the hands and objects are well reconstructed in real data, and with the help of tactile information, we can reconstruct the detailed shape occluded by the hand (Supplementary Movie 3). More importantly, the reconstructed deformable sponges based on the tactile feedback after strain interference suppression could show more negligible details in the region with obvious strain, and the completeness of the rigid object obtains improvements because the method of strain interference suppression help to recover the real small forces exerted on the rigid edges. In Fig. 6B, we present the gradually deformed plasticine that represents a dumpling-making task of pinching the dumpling wrapper. The deformation of the plasticine in every step has been well demonstrated. In Fig. 6C, the reconstruction of a rigid folding rack is achieved, which employs multiple contacts by the hand on different spots on the object. The details of the folding rack have been completed through multiple contacts with procedural tactile embedding. Moreover, to demonstrate the necessity of visual-tactile joint learning, we present visual-only and visual-and-tactile results for a scissor, a folding rack, and a bottle in Fig. 6D and Supplementary Movie 4. Benefiting from the combining visual and tactile features, both the rigid and deformable objects have been well reconstructed. In Fig. 6E and Supplementary Movie 5, the well-reconstructed sequences prove our method is capable of handling sequential data with multiple frames. Therefore, the improved performance of our visual-tactile model proves that it is crucial to introduce tactile information with strain interference suppression, both for obtaining the feature occluded by the hand and for acquiring the dynamic deformation of the objects on the stretchable interfaces.

To validate the effectiveness of our proposed ViTaM system, qualitative and quantitative comparative tests have been conducted to answer the following questions: (1) whether the tactile array-specific data format is effective in conveying the geometry information to the learning algorithm? (2) whether it is more effective compared to other forms of sensors, such as RGB-D cameras, or optical tactile sensors? In experiments, first, we compared the performance of existing vision-only solutions with the algorithm of the ViTaM system excluding the tactile encoder70,71; second, the algorithm is compared with a previous work, VTacO72, which employs a gel-based optical tactile sensor, DIGIT, to record contact deformation. The comparison of chamfer distance is shown in Supplementary Table 7. Due to the poor wearability and flexibility of the VTacO system, it was only used for two simple actions: touching and grasping objects (Fig. S26). The results are shown in Fig. S27, Supplementary Table 8, and Supplementary Table 9. It can be found that the ViTaM system demonstrates superior performance over purely visual methods in reconstructing four types of objects: elastic, plastic, articulated, and rigid. For example, the chamfer distance of reconstructing a sponge using the ViTaM system is only 0.467 cm, which is an improvement of 36% compared to that of VTacO. While gel-based optical sensors can obtain higher-resolution local geometry information, such as sharp edges or severe deformation, our distributed design of the tactile glove can obtain more comprehensive features when the occlusion is too severe for the vision information.

To validate the design of the algorithm, we ablate the temporal transformer module in the temporal feature fusion part. For testing the performance without it, we only use a self-attention module to encode the fused feature and use the same WNF Predictor to decode the winding number values. The results are shown in Supplementary Table 10. They indicate that the introduction of the temporal transformer module significantly enhances performance, likely due to the consistency of inter-frame features within a video sequence and the force differences between frames, which provide information about object deformation or changes in joint states. In summary, the tactile glove with good wearability and the accompanying algorithm exhibits significant improvements over existing vision-based tactile sensors, ensuring superior operational flexibility in hand-object interaction tasks and demonstrating obvious advantages in reconstruction results. Besides, we report the performance over the video sequence to validate the temporal consistency. The results are given in Supplementary Table 11. The “Per-frame” row reports the mean per-frame error for the entire category. While the “full-video” row reports the chamfer distance calculated by first average on a full video, then on a whole category. Due to the difference in frame numbers between different manipulation processes, these two metrics have different scores. The variance in the full-video metric is also reported. Additionally, we tested the model’s average inference runtime on one Nvidia RTX 4090 GPU, which is around 3-5 frames per second.

The extensive experiments have proved the efficacy of the proposed algorithm on the compatibility to the glove, the superiority over the baseline methods, and the consistency of dynamic reconstruction.

Methods
Preparation of the tactile sensing block
Similar to our previous research65, the multi-walled CNT aqueous dispersion (XFZ29, Nanjing XFNANO Materials Tech Co. Ltd., China; length of the CNTs ≤10 μm) and the natural latex solution (001a, Maoming Zhengmao Petrochemical Co. Ltd., China; ammonia content: 0.2%) are pre-dispersed by ultrasound for 15 min in ice bath. They are respectively mixed in the ratio of 1 to 1 (equal to 5 wt% CNT content) and 1 to 2.5 (equal to 2.9 wt% CNT content) to fabricate the negative and positive effect membrane. The mixtures with 5 wt% and 2.9 wt% CNT content are magnetically stirred for 6 h and 9 h in the ice bath, respectively. Then, the above mixtures are all molded by drop-casting and cured for 15 h, followed by being cut into rectangular shapes with a size of 10 mm × 20 mm × 0.4 mm. After releasing from the molds, sew the conductive fabric wires that serve as electrodes at both ends of each membrane to monitor the strain variation. The resistivity of the conductive fabric wire is 1–2 Ω·cm. The negative and positive effect membranes are assembled with non-conductive adhesive. These two pairs of electrodes are respectively connected to the strain interference detection circuit.

Fabrication of the tactile glove
For every piece of the block shown in Fig. S8A, four-row electrodes, and three-column electrodes are vertically sewn onto the block, sequentially. The spaces between row and column electrodes are set to 2.5 mm and 3 mm, respectively. The wire end of each electrode is fixed by UV curing adhesive to prevent shedding. For the index, middle, ring, and little fingers, there is a tactile sensing block on each of the three knuckles of each finger, whereas there are two tactile sensing blocks on the two knuckles of the thumb. So, the number of tactile sensing blocks on the fingers is fourteen. Tactile sensing blocks on the same finger share column electrodes. The tactile sensing blocks on the palm are arranged in a 4 × 6 array. All the tactile sensing blocks share the row electrodes. To stably encapsulate the conductive fabric wires and the pads on the FPC, the conductive fabric wire is first fixed on the pad by tying a knot, then fixed with low-temperature fast-drying conductive silver paste, and then finally encapsulated with UV curing adhesive.

Secondly, there are three types of FPCs: finger FPCs and a palm FPC connecting the finger blocks and palm blocks with the multi-channel scanning circuit; the general FPCs connecting the scanning circuit and the processing circuit. The first two kinds of FPCs that sewed on the fabric glove by fixing holes have electrical pads with exposed deposited gold on them. The FPCs are fabricated by typical processes including copper electroplating on the polyimide (PI) layer, dry film lamination, exposure, developing, etching, dry film stripping, cover PI layer lamination, laser cutting, and assembling.

Third, the tactile glove is integrated with a modular assembly process. The tactile sensing blocks are fixed on the fabric glove by a handmade sewing process. The miniaturized multi-channel scanning circuit is fixed on the back of the glove hand. The processing circuit is fixed in the middle of the bracelet.

Hardware implementation and simulation of the scanning circuit
Based on the multiplexing technology of electrodes, this scanning circuit can realize row-by-row and column-by-column scanning of 24-row electrodes by 48-column electrodes for a total of 1152 tactile sensing units. There are 6 sample modules in this current-mode scanning circuit, in which every sample module has one 8-channel ADC chip (AD7928, Analog Devices Inc., USA) and two 4-channel amplifier chips (ADA4691-4, Analog Devices Inc., USA). Every column electrode is connected to an amplifier. By matching the resistor Rg with the tactile sensing unit, gain-adjustable amplification of tactile signals can be achieved. To perform a cyclic scanning read of the row electrodes, two 4-to-16-line decoders are used. For 24 rows of electrodes, only one electrode is grounded at a time, while the other unselected electrodes remain connected to the reference voltage. The voltage at the positive input of the amplifier also needs to be set to the reference voltage, which is as same as the voltage at the unselected row electrodes, so no leakage current flows at the unselected row electrodes. The multi-channel scanning circuit also includes two sets of independent power supplies to ensure that the multi-channel scanning sampling design has sufficient driving capacity. In addition, the processing circuit not only connects the column electrodes of the tactile sensing blocks on the palm range with the multi-channel scanning circuit but also includes a microcontroller unit (MCU) (ESP32, Espressif Systems (Shanghai) Co., Ltd., China) and communication modules for data processing and transmission.

Electrical and mechanical characterization
The tactile sensing block is sequentially stretched to different strains from 5% to 30% and fixed on the Teflon substrate. The tensile tester (AGS-X50N, SHIMADZU) equipped with a customized probe is used to apply normal force. Meanwhile, the resistance changes of the tactile sensing unit are measured by an LCR meter (4100, Wayne Kerr). To validate the robustness of the tactile sensing block, a dynamic thermomechanical analysis machine (Q850, TA Instruments) is employed for the repeatability test. The output voltage of the tactile sensing block is collected by the proposed circuit.

The static calibration of the tactile sensing blocks
The tactile sensing blocks of the tactile glove are placed on the test platform made by PDMS elastomer (Sylgard-184, Dow Corning), which has a similar elastic modulus to human skin. All of the tactile sensing blocks have been calibrated with a force interval of 0.05 N by the tensile tester (AGS-X50N, SHIMADZU) from 0 N to 40 N. The voltage changes of the tactile sensing unit are directly read out from the computer synchronously. The relationship between voltage change and loading force is determined through a mathematical fitting model (Supplementary Note 1). Testing of all modules is performed 4 times and averaged to minimize tester errors. Mechanical exception values of some tactile sensing units are multiplied by a calibration factor to obtain a relatively smooth calibration curve.

System integration of the hardware and VR environment
The high-density tactile glove generates a frame of 24 × 48 tactile data, which is sent to a personal computer (PC) via a standard serial port. The other end of the serial port is connected to a PC running the real-time visualization program. The visualization program is comprised of two components: an interactive RFUniverse-based interface, and a parser that decrypts serial port data. The parser continuously monitors the incoming data from the serial port; upon detecting a header, it interprets the subsequent data as a frame and verifies it against the checksum. Although inconsistencies between data frames and their checksums are rare, the parser discards any non-compliant data frames automatically, which ensures data integrity. Frames with correct checksums are converted back into a two-dimensional matrix by the parser. During the restoration process, a pre-calibrated voltage-to-pressure mapping is applied, translating voltage readings back into pressure measurements. This two-dimensional matrix is later transmitted to the RFUniverse interface with RFUniverse’s communication protocol. Through this interactive interface, users can interact with a user-friendly GUI to observe the status of the glove intuitively. Following these procedures, the pressure values of the glove are consistently gathered and demonstrated animatedly and in real-time. If necessary, the parser simultaneously copies the data stream to local storage for further in-depth utilization.

Data collection in simulation and real-world
In simulation, we generate a random pose with a human hand model to interact with an object. Both hand and object are modeled by FEM. The contact is modeled by incremental potential contact (IPC)71 to guarantee mesh intersection-free and inversion-free during dynamics simulation. The simulation environment to generate training data is supported by RFUniverse. We run RFUniverse on a PC with an Intel i9 13900K CPU, NVIDIA GTX 4090 Graphics card, and 64 GB memory. The runtime for generating a data sample is 0.5 fps.

In the real world, we collect five objects of the same category but unseen instances from the training set. The operator wears the tactile glove and sits in front of a table. A top-down RGB-D camera is set to record the manipulation process. For objects with elasticity and rigidity, we apply grasping and pinching manipulation. After the manipulation is done, we present the in-contact object to the camera. For objects with plasticity, such as the plasticine, we apply multi-step manipulation. After each manipulation, we move the hand away from the object so that the object can be directly presented to the camera. When labeling the data, we manually remove the non-object point cloud from the RGB-D camera. The RGB-D camera to record the test data in the real world is the Photoneo MotionCam M+ model. It is placed 1.55 m high up and straight down to the table and records the data at 10 fps with an 800 × 1120 RGB stream and 800 × 1120 depth stream.

Reconstruction and tracking of highly non-rigid objects
The visual-tactile learning model is adapted from our previous work VTacO71. VTacO processes the regular-format tactile image as the tactile representation, while here we process the tactile reading as an 1152-d vector. Thus, we replace the tactile encoder in VTacO with a 3-layer MLP and a self-attention module and divide the tactile reading into 38 blocks to compute the tactile feature. The contact positions are inferred by the positions of every sensor block on the glove, and block positions are predicted through hand shape estimation. In addition, we recast the visual-tactile reconstruction framework into a tracking pipeline, so that it can take multiple frames of visual-tactile images as input and give reconstruction for each frame with spatial-temporal consistency. The detailed structure and implementation details are described and illustrated in Supplementary Note 2-4.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "In this article, the authors proposed a visual-tactile recording and tracking system (ViTaM), which includes a stretchable tactile glove and a visual-tactile joint learning framework to estimate dynamic hand-object states during manipulation. The authors validated their findings by computing the accuracy in force measurement and the shape reconstruction error for both deformable and rigid objects.\nAs my expertise is in artificial intelligence, not hardware design, I'll primarily focus on the learning framework. I think the proposed learning framework is not novel, and surprisingly, comparisons with existing methods are lacking. In this case, it becomes impossible to assess whether the proposed framework has an advantage, either scientifically or empirically, over prior art in this area.\nSpecifically, in the supplementary material, the authors summarized a number of existing methods that perform visual-tactile reconstruction of object geometry. These methods mostly build on publicly available sensors, such as GelSight and DIGIT. Thus, it seems a natural and required step to replicate and compare with some of these systems (with their released code) in order to demonstrate the superiority of the proposed shape reconstruction methods.\nWithout such comparison, we may only assess the reconstruction quality based on qualitative results. In addition to methods that take multiple sensory inputs, there has been fast progress in vision-based shape reconstruction of hand-object interaction [1, 2, 3], including some from the authors' institution [2] or from the authors themselves [3]. The authors should also compare the proposed system with these methods and explain the superiority of the proposed joint learning framework.\n[1] Chen et al. Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild. AAAI 2023\n[2] Li et al. CHORD: Category-level Hand-held Object Reconstruction via Shape Deformation. ICCV 2023\n[3] Xu et al. Visual-Tactile Sensing for In-Hand Object Reconstruction. CVPR 2023\nThe submission's presentation is fine. I have two suggestions: the first is that there are places where it feels unclear whether ViTaM refers to the learning framework only or the entire system (hardware and software together). To avoid confusion, it would be beneficial to clarify whether ViTaM is a comprehensive system that includes both the learning framework and the hardware components.\nThe second and more important issue is that the introduction is written in a confusing way. It spends much space on the challenges in hardware design without proposing a solution. When it finally introduced ViTaM as a solution to these challenges, it became hard to understand the motivation behind all design choices and how they addressed the challenges listed in the very long, definitely-too-long second paragraph of the introduction. I believe the introduction would benefit from a rewrite.", "reviewer_2": "[Summary]\nThis paper introduces a tactile glove and a visual-tactile joint learning framework named ViTaM. The glove is equipped with 1152 sensing channels, and the authors employ a strain interference suppression technique that achieves high accuracy in force measurement, improving measurement accuracy by 45.3% over uncalibrated methods.\nThe authors collected multimodal data containing both visual and tactile information, utilizing the ViTaM framework for several downstream tasks. Notable ones among these tasks are the reconstruction of in-hand objects using both vision and touch, and the tracking and reconstruction of dynamic hand-object interactions.\n[Strengths]\nThe introduced new tactile glove is interesting and the authors have integrated more than a thousand tactile sensing channels. The authors have also provided an analysis of the signal correlating the sensing response and the external force. A calibration process is also in place to enhance the quality of the signal.\nLeveraging the tactile glove, the authors have also tackled the challenging tasks of in-hand objects and the tracking/reconstruction of dynamic hand-object interactions. The authors have shown some quite impressive reconstruction/tracking results, especially examples involving deformable objects like sponges and plasticine.\n[Weaknesses]\nWhile I appreciate the general direction of this work, it is crucial to provide a more thorough analysis of the sensor characteristics and calibration design. Additionally, a more detailed and comprehensive analysis of the experimental results is necessary.\n** Questions regarding sensor characteristics **\nFrom the dumpling-making video, there appears to be significant sensor noise, including regions that are not in contact but still blink and show a large force on the color bar.\nPrevious works have demonstrated continuous reading of the tactile signal [1], but in this tactile glove, the sensing is only discrete and categorized into only five categories.\nThe color bar is also somewhat confusing; using different colors for different force scales makes it difficult to intuitively assess the actual force. Would a continuously changing color be more intuitive?\nDuring the kneading and pressing motions, I was expecting a more continuous and smooth transition of the contacting force patterns. However, the visualization shows that the sensing patterns are very discrete and noisy. There are many gaps (e.g., sensors not activated) within regions that are clearly in contact.\nAlso, when using the fingertips to press the dough, the signals from the fingertip regions are not particularly obvious. \nThere is a lack of detailed information regarding the frame rate used during the tactile glove’s operation, which is crucial for evaluating the glove's responsiveness and applicability in real-time applications.\nI noticed that many sensors have never lit up; have the authors checked whether there might be any dead sensors?\nThe noise from the sensor is also evident from Fig. 3. Characterizations of the tactile array. I: even when the strain is at 5%, the error rate is already 20%.\nI am also curious about the interference from the hand pose. Will different hand poses also lead to different tactile patterns and, if so, how to disentangle signals from the hand pose and the actual physical interactions with the objects?\n“The repeatability of the tactile array under a load pressure of 5 kPa over 2000 times,”—what about drifting after extensive use with human hands?\nI’m also curious about the possibility of mass-producing the glove.\n** Questions regarding the calibration process **\nGiven the limited quality of the obtained tactile signal, a more comprehensive design and analysis of the calibration process is necessary.\nThe authors claim that “The relationship between voltage change and loading force is determined through a mathematical fitting model.” A more detailed explanation is needed. For example, how much data and what kind of data are used? Which mathematical fitting model is used, and how well does it generalize? Are the generalization capabilities different for various regions of the hand and during different interactions with the environment? What about the generalization capability when interacting with objects of different materials?\n** Questions regarding the joint learning framework **\nWhile the authors have conducted ablation studies comparing models taking both vision and touch and their counterparts that take either vision or touch as input, only three qualitative examples are shown. For a more comprehensive understanding of the performance, it is essential to demonstrate the benefit of the tactile glove through more comprehensive qualitative and quantitative comparisons.\nRelated to my previous point, it is also crucial to conduct a much more comprehensive study on the model’s performance for the dynamic reconstruction of hand-object interactions.\nThe authors claim that their framework enables “the tracking and geometric 3D reconstruction of manipulated arbitrary objects.” This significant claim requires significant evidence. The authors should be more concrete about what type of objects their framework can reconstruct (e.g., rigid, elastic, plastic, or articulated) and provide sufficient evidence to justify the claim.\nFor the hand-object reconstruction, there appear to be a lot of gaps between the hand and the object, especially in the scissor example. Would priors like contact, force, or form closure be helpful here?\nIt seems that the hand pose is estimated from the point cloud. Will the cables on the glove and the glove itself interfere with the pose estimation results? Meanwhile, is it possible for the glove to directly provide the ground truth hand pose, either through the obtained tactile signal or some other integrated sensors?\nFor the dynamic reconstruction, both hand and object are modeled by FEM, and the simulation environment to generate training data is supported by RFUniverse. It would provide readers with a more comprehensive understanding if the authors could provide more details and visualizations of the simulation.\nIt would also be helpful to provide a quantitative measurement of how the simulation matches the data collected in the real world, especially considering the noise from the sensors.\n[1] Sundaram et al., Learning the Signatures of the Human Grasp Using a Scalable Tactile Glove, Nature 569, 698–702 (2019)."}, "author_response": "Response Letter\nReviewer #1 (Remarks to the Author):\nIn this article, the authors proposed a visual-tactile recording and tracking system\n(ViTaM), which includes a stretchable tactile glove and a visual-tactile joint learning\nframework to estimate dynamic hand-object states during manipulation. The authors\nvalidated their findings by computing the accuracy in force measurement and the shape\nreconstruction error for both deformable and rigid objects.\nAuthors:\nThank you for taking the time to review our manuscript and provide valuable\nfeedback and suggestions. We have carefully considered and analyzed all of your\ncomments. We have summarized the issues based on the sections of your review\ncomments and provided detailed responses to each point.\n1. As my expertise is in artificial intelligence, not hardware design, I'll primarily focus\non the learning framework. I think the proposed learning framework is not novel, and\nsurprisingly, comparisons with existing methods are lacking. In this case, it becomes\nimpossible to assess whether the proposed framework has an advantage, either\nscientifically or empirically, over prior art in this area.\nSpecifically, in the supplementary material, the authors summarized a number of\nexisting methods that perform visual-tactile reconstruction of object geometry. These\nmethods mostly build on publicly available sensors, such as GelSight and DIGIT. Thus,\nit seems a natural and required step to replicate and compare with some of these systems\n(with their released code) in order to demonstrate the superiority of the proposed shape\nreconstruction methods.\nAuthors:\nWe appreciate your guidance on our learning framework from the perspective of\nan artificial intelligence expert. We are aware that comparing with “existing” algorithms\nis important. However, compared to our work, other types of tactile sensors exhibit\nsignificant differences in terms of input and output, task types, and algorithm\nframeworks. The following sections will provide a detailed analysis of these differences.\nFirst of all, input and output are the core factors to consider in designing a neural\nnetwork pipeline. For the input, in our framework, the input is highly related to the\nproposed tactile array, which does not have similar open-source competitive products\nto compare with. Some open-source sensors such as DIGIT are optical sensors, that\ncreate tactile ability in a way that is fundamentally different from piezoresistive sensors\nlike ours. The direct differences in the input are the data format and content. The tactile\nimages from optical sensors reflect the deformation of the contacted gel at the fingertip,\nwhich is local and requires a 3D reconstruction step to obtain the deformation, finally\nobtains the force reading by calculating the linear relationship between the deformation\nand force load based on pre-calibrated material parameters. The tactile arrays from our\nglove reflect the contact states (e.g. contact or not, force reading, etc.) all over the hand\npalm.\nThe tactile image has a higher resolution, typically 512 × 512, but has a lot of\nredundancy and is non-trivial to obtain accurate forces since the geometry estimation\nfrom tactile images is still an open problem1,2. In contrast, our multiple-channel glove\ncan directly read the force measurement with an accuracy of 97.6%. Thus, the\ndifference in data features will significantly change the way we process the data. Since\nthey contain different levels of information.\nFrom the output side, dynamic reconstruction of deformable objects has long been\ndiscussed in the computer vision community3,4. However, most previous works cannot\ngeneralize well to different deformable objects, not to mention compatible with\ndifferent object types (i.e. rigid, articulated, or deformable). Therefore, we believe our\ntask of dynamic reconstruction on different object types is novel in task settings.\nAnother factor to consider is the module arrangement in neural network design. To\nfacilitate the reconstruction in a dynamic setting, we adopt the cross-attention\nmechanism to fuse inter-frame features. It is also significantly different from the VTacO\ndesign.\nIn summary, the novelty of the reconstruction framework lies in two places: the\ntask setting on both input and output, and the introduction of dynamic attention.\nTherefore, the comparison with the baseline methods has also been discussed separately.\nThe novelty of the task setting originated from the glove design. Thus, we compare\nthe reconstruction performance with a different sensor, DIGIT, as suggested by the\nreviewer. With DIGIT, the encoder and decoder are the same as VTacO (reference [3]\nmentioned by the reviewer).\nSince the optical sensors are relatively thick and can’t reach high wearability and\nconformality (Fig. R1-1(A)), we wear them on the hands and demonstrate two types of\nhand poses for reconstruction: touching and grabbing (Fig. R1-1(B)). The quantitative\nresults are shown in Table R1-1 and Table R1-2. The smaller the chamfer distance, the\nbetter the performance. The last row of the Table provides a comparison of the\nperformance between ViTaM and VTacO. If the difference between the two is less than\n0.1 cm, it is called “On par”, otherwise it is judged as “Better” or “Inferior”.\nFig. R1-1. (A) The front view, back view, and side view of the optical sensors with poor\nwearability and conformality when they are fixed on the fingers; (B) Two types of hand poses\n(touching and grabbing) demonstrated by the above optical sensors\nTable R1-1:\nThe comparison between VTacO and ViTaM using touching poses.\nObj. Type Elastic Plastic Articulated Rigid\nChamfer Folding\nCup Bottle Sponge Plasticine Scissor\nDistance (cm) rack\nVTacO 0.937 0.945 0.546 2.314 1.002 1.534\nViTaM\n0.921 0.953 0.503 2.429 0.991 1.549\n(this work)\nPerformance\nBetter On par Better Inferior Better On par\nComparison\nWe can see from Table R1-1 and Table R1-2 that, although the two algorithms reach\nsimilar performances with hand-touching objects, ViTaM has outperformed VTacO in\nthe case of hand-grabbing objects. We figure that while gel-based sensors can obtain\nhigher-resolution local geometry information, such as sharp edges or severe\ndeformation, our new distributed design of the tactile glove can obtain more\ncomprehensive features when the occlusion is too severe for the vision information.\nSince hand-object interactions are more likely to involve complex manipulation of the\nobject, inevitably leading to more severe occlusions. Our ViTaM system and the\nproposed pipeline are a better choice for hand-object reconstruction. We also\ndemonstrate some qualitative results in Fig. R1-2.\nTable R1-2:\nThe comparison between VTacO and ViTaM using grabbing poses.\nObj. Type Elastic Plastic Articulated Rigid\nChamfer Folding\nCup Bottle Sponge Plasticine Scissor\nDistance (cm) rack\nVTacO 1.125 0.995 0.733 3.245 1.103 1.509\nViTaM\n0.954 0.985 0.467 2.106 0.925 1.501\n(this work)\nPerformance On On\nBetter Better Better Better\nComparison par par\nFig. R1-2. Qualitative results for comparing ViTaM-Glove and the DIGIT in the system of\nVTacO\nAnother aspect of the experiments we have done is to examine the effect brought\nby the temporal transformer module in the temporal feature fusion part. For testing the\nperformance without it, we only use a self-attention module to encode the fused feature\nand use the same WNF Predictor to decode the winding number values. The results are\nshown in Table R1-3.\nThe results show that with the introduction of a temporal transformer module,\nsignificant performance gets boosted. It is probably because of the consistency of the\ninter-frame features in one video sequence and the force difference in two frames\ncontains information about the object deformation or joint state changes.\nTable R1-3:\nThe error with or without temporal transformer (TT) module.\nObj. Type Elastic Plastic Articulated Rigid\nChamfer\nFolding\nDistance Cup Bottle Sponge Plasticine Scissor\nrack\n(cm)\nViTaM\n0.953 0.984 0.435 2.075 0.903 1.451\n(w TT)\nViTaM\n1.035 1.112 0.631 2.431 0.975 1.459\n(w/o TT)\nIn summary, the tactile glove and its ViTaM system proposed in this paper exhibit\nsignificant improvements over existing vision-based tactile sensors from both\nquantitative and qualitative perspectives. Its excellent wearability ensures superior\noperational flexibility in hand-object interaction tasks, and it also demonstrates obvious\nadvantages in reconstruction results.\nThe corresponding results have been added as supplementary information. Thank\nyou once again for your question!\nReferences:\n1. Yuan, W., Dong, S. & Adelson, E. Gelsight: high-resolution robot tactile sensors for estimating\ngeometry and force. Sensors 17, 2762 (2017).\n2. Wang, S. et al. GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a\nCompact Robot Finger. in Proceedings of the 2021 IEEE International Conference on Robotics\nand Automation (ICRA) 6468-6475 (IEEE, Xi’an, China, 2021).\n3. Li, K. et al. Chord: Category-level Hand-held Object Reconstruction via Shape Deformation. in\nProceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV) 9410-\n9420 (IEEE/CVF, Paris, France, 2023).\n4. Tsoli, A. & Argyros, A. A. Joint 3D Tracking of a Deformable Object in Interaction with a Hand.\nin Computer Vision – ECCV 2018 (eds. Ferrari, V., Hebert, M., Sminchisescu, C. & Weiss, Y.)\nvol. 11218, 504–520 (Springer International Publishing, Cham, 2018).\n2. Without such comparison, we may only assess the reconstruction quality based on\nqualitative results. In addition to methods that take multiple sensory inputs, there has\nbeen fast progress in vision-based shape reconstruction of hand-object interaction [1, 2,\n3], including some from the authors' institution [2] or from the authors themselves [3].\nThe authors should also compare the proposed system with these methods and explain\nthe superiority of the proposed joint learning framework.\n[1] Chen et al. Tracking and Reconstructing Hand Object Interactions from Point Cloud\nSequences in the Wild. AAAI 2023\n[2] Li et al. CHORD: Category-level Hand-held Object Reconstruction via Shape\nDeformation. ICCV 2023\n[3] Xu et al. Visual-Tactile Sensing for In-Hand Object Reconstruction. CVPR 2023\nAuthors:\nThank you for this professional question! We have compared with these hand-\nobject reconstruction baseline as you mentioned, and the results are reported in Table\nR1-4.\nTable R1-4:\nThe comparison between different baselines and our work.\nObj Type Elastic Plastic Articulated Rigid\nChamfer\nFolding\nTrack Distance Input Cup Bottle Sponge Plasticine Scissor\nrack\n(cm)\nCHORD RGB 1.005 1.031 0.794 2.883 1.239 1.598\nFull\nVision-\nHOTrack Segmented 0.998 1.001 0.596 2.723 0.954 1.575\nonly\nPC\nViTaM Partial\n(Vision- unsegmented 0.989 1.015 0.631 2.881 1.023 1.562.\nonly) PC\nPartial\nunsegmented\nVTacO 1.031 0.980 0.619 2.675 1.005 1.510\nPC+Tactile\nVisual- Image\nTactile Partial\nunsegmented\nViTaM 0.953 0.984 0.435 2.075 0.903 1.451\nPC+ Tactile\nArray\nOn\nPerformance Comparison Better Better Better Better Better\npar\nFor CHORD [2] and HOTrack [1], these two methods are vision-only input. Thus,\nto compare with them, we remove the tactile encoder in the algorithm of ViTaM,\ndenoted “ViTaM (Vision-only)”. It is noteworthy that, even though these baseline\nmethods are vision-only, their inputs are of different data formats. Different data\nformats can provide different levels of information, which also influences the results.\nCHORD extracts features from 2D images and optimizes the geometry via\ndifferentiable rendering, which can produce nice-looking reconstruction results but\ncannot capture the actual object dimensions. Besides, the RGB-based method is highly\nvulnerable to the occlusion. Thus the performance of CHORD is compromised.\nHOTrack takes in full and separated point clouds of hand and object, which actually\nreduces the influence of occlusion and self-occlusion from the observer’s perspective,\nand makes the neural network easier to process hand and object individually. Therefore,\nit has the best performance on the vision-only track. The input provides the most\ncomplete information. However, for the ViTaM (Vision-only) model, it takes in a partial,\nunsegmented point cloud, which is closer to real-world application scenarios.\nFor VTacO [3], we attach the DIGIT sensors on the hand and perform grasping on\nthe object. Since the DIGIT cannot recover most object surfaces, VTacO does not\nconsider the inter-frame feature fusion. The reconstructed results from VTacO are less\ncomplete and consistent between frames.\nThe corresponding results have been added as supplementary information. Thank\nyou!\n3. The submission’s presentation is fine. I have two suggestions:\nThe first is that there are places where it feels unclear whether ViTaM refers to the\nlearning framework only or the entire system (hardware and software together). To\navoid confusion, it would be beneficial to clarify whether ViTaM is a comprehensive\nsystem that includes both the learning framework and the hardware components.\nAuthors:\nWe apologize for any unclear statements about the ViTaM in our manuscript. In\nfact, \"ViTaM\" is a comprehensive system comprising both hardware and software,\nwhich are tightly coupled and work together to achieve the application examples\npresented in the manuscript. In the name “ViTaM”, “Vi” represents visual perception,\n“Ta” stands for tactile perception, and “M” means manipulation. The hardware includes\na glove-shaped stretchable tactile array for recording force interaction, and the software\nrefers to a deep learning framework for visual-tactile data processing and hand-object\nstate estimation during manipulation. In the manuscript, we use “ViTaM-Glove”, and\n“the learning algorithm of ViTaM system” to discern the two components.\nWe have revised the descriptions in the manuscript and highlighted them in red to\nensure consistency throughout the text.\n4. The second and more important issue is that the introduction is written in a confusing\nway. It spends much space on the challenges in hardware design without proposing a\nsolution. When it finally introduced ViTaM as a solution to these challenges, it became\nhard to understand the motivation behind all design choices and how they addressed\nthe challenges listed in the very long, definitely-too-long second paragraph of the\nintroduction. I believe the introduction would benefit from a rewrite.\nAuthors:\nThank you for your reminder! We have revised the introduction based on your\nsuggestions and highlighted them in red. The second paragraph has also been condensed\naccordingly. In the new version, we focus on the challenges of traditional tactile signal\nacquisition and provide a comparative explanation, aiming to help readers better\nunderstand our approach.\n5. Reviewer #1 (Remarks on code availability):\nThe code looks reasonable, but is heavily dependent on another published paper.\nThe reviewer cannot run the code due to the lack of required hardware.\nAuthors:\nThank you for your feedback. We apologize for any inconvenience caused. We\nactually restructured the codebase of VTacO to make it adaptive to ViTaM. Since we\nplan to make the codebase suits different input sources and different neural network\narchitecture designs, thus it looks like the codebase is heavily dependent on VTacO. In\ncomparison with VTacO, we have added a new encoder for a glove-like tactile array,\nand a temporal transformer module for inter-frame feature fusion.\nTo avoid such unnecessary misunderstanding, we provide a standalone codebase\nfor ViTaM and a small sample to allow reviewers to run it quickly. The repository link\nstays the same: https://github.com/jeffsonyu/ViTaM.\nReviewer #2 (Remarks to the Author):\n[Summary]\nThis paper introduces a tactile glove and a visual-tactile joint learning framework\nnamed ViTaM. The glove is equipped with 1152 sensing channels, and the authors\nemploy a strain interference suppression technique that achieves high accuracy in force\nmeasurement, improving measurement accuracy by 45.3% over uncalibrated methods.\nThe authors collected multimodal data containing both visual and tactile\ninformation, utilizing the ViTaM framework for several downstream tasks. Notable\nones among these tasks are the reconstruction of in-hand objects using both vision and\ntouch, and the tracking and reconstruction of dynamic hand-object interactions.\nAuthors:\nFirst and foremost, we would like to express our sincere gratitude for taking the\ntime to review our manuscript and providing us with such valuable and professional\nfeedback from several perspectives. We greatly appreciate your insightful comments\nand have carefully considered and tried our best to address each of them. Below, we\nprovide our detailed responses and the corresponding revisions made to the manuscript.\n[Strengths]\nThe introduced new tactile glove is interesting and the authors have integrated more\nthan a thousand tactile sensing channels. The authors have also provided an analysis of\nthe signal correlating the sensing response and the external force. A calibration process\nis also in place to enhance the quality of the signal.\nLeveraging the tactile glove, the authors have also tackled the challenging tasks of\nin-hand objects and the tracking/reconstruction of dynamic hand-object interactions.\nThe authors have shown some quite impressive reconstruction/tracking results,\nespecially examples involving deformable objects like sponges and plasticine.\nAuthors:\nThank you for your positive feedback on our research work, recognizing it as both\ninteresting and impressive. We have made concerted efforts in both hardware and\nsoftware to address the challenging tasks of in-hand object tracking and reconstruction.\n[Weaknesses]\nWhile I appreciate the general direction of this work, it is crucial to provide a more\nthorough analysis of the sensor characteristics and calibration design. Additionally, a\nmore detailed and comprehensive analysis of the experimental results is necessary.\nAuthors:\nAppreciate your comments. We have conducted an in-depth and comprehensive\nanalysis of the sensor characteristics and calibration design in accordance with your\nsuggestions.\n** Questions regarding sensor characteristics **\n1. From the dumpling-making video, there appears to be significant sensor noise,\nincluding regions that are not in contact but still blink and show a large force on the\ncolor bar.\nAuthors:\nWe thank the reviewer for bringing this to our attention. In the video of performing\nthe task of making dumplings, there were indeed some noticeable noise points, as\nshown in Fig. R2-1. However, these noise points were concentrated around the base of\nthe fingers, close to the palm. Due to the wearer’s thicker finger bases, wearing this\ntactile glove causes the base area to be relatively tight, which exerts pressure on the\nsensors, resulting in some noise points in the image. Additionally, the finger bases tend\nto press against the palm during actions, creating some flickering noise points. When\nthe tactile gloves are placed horizontally on the desktop so that they are not affected by\nexternal forces, no noise points can be seen, as shown in Fig. R2-2.\nFig. R2-1. The areas where noise commonly occurs during the three actions of making dumplings:\n(a) kneading, (b) pressing, and (c) pinching.\nFig. R2-2. The visualized interface without noise when the tactile glove is placed horizontally on\nthe desktop\nDuring the process of hand-object interaction, the occurrence of noise is\nunavoidable; the key is to assess how much the noise affects the effective signal. Based\non the analysis above, it can be observed that the locations and values of these noise\npoints are relatively fixed. Moreover, the hand-object reconstructions are completed by\nthe fusion information of tactile and visual, which means the visual information can\nemphasize where are the main contact areas, thus helping to focus the attention on these\nareas. Therefore, during dynamic operations, we can ignore the interference of these\nnoise points by the deep-learning algorithm, and this will not have a concerning impact\non the dynamic capturing ability of the tactile glove. In the future, the selection of\ngloves can be adjusted according to the user’s preferences, and suitable sizes can be\ncustomized to reduce unnecessary noise.\n2. Previous works have demonstrated continuous reading of the tactile signal [1], but in\nthis tactile glove, the sensing is only discrete and categorized into only five categories.\nAuthors:\nThanks for kindly reminding us to clarify this point. It is clear that our proposed\ntactile glove is also capable of continuously collecting tactile force signals (similar to\nthe continuous reading described in [1]), rather than just five categories. The reason for\nthis ambiguity is that we used five color blocks in the visualization interface to represent\ndifferent force ranges. We apologize for this confusion. Fig. R2-3(a) displays the python\ncode of the color bar settings in this work and it can be found that the measurement\nrange of the force covers the entire scale from 2.5 V to 5 V. In Fig. R2-3(b), the voltage\nranges and the corresponding colors are offered. Besides, Fig. R2-3(c) shows the final\nvoltage output curve of the tactile sensing unit versus the loaded forces or pressures,\nwhich can also directly indicate that the sensing of tactile signals is continuous rather\nthan discrete.\nFig. R2-3. (a) Python code of the color bar settings; (b) The color bar with corresponding voltage\nranges; (c) The relationship of voltage output curve of the tactile sensing unit and the loaded\nforces or pressures\n3. The color bar is also somewhat confusing; using different colors for different force\nscales makes it difficult to intuitively assess the actual force. Would a continuously\nchanging color be more intuitive?\nAuthors:\nWe agree with the reviewer’s point. The color bar has been reset to show a\ncontinuous color change when the forces are loaded. Fig. R2-4 shows the tactile sensing\nresults with the updated color bar. Additionally, all relevant visualization images in the\nmanuscript have been updated.\nFig. R2-4. The visualized interface features a color bar that continuously changes shades of red\n4. During the kneading and pressing motions, I was expecting a more continuous and\nsmooth transition of the contacting force patterns. However, the visualization shows that\nthe sensing patterns are very discrete and noisy. There are many gaps (e.g., sensors\nnot activated) within regions that are clearly in contact.\nAuthors:\nThank you for your professional review. Based on the issues you identified earlier,\nwe have made corrections to the noise and color bar of the tactile gloves. On the one\nhand, this can reduce the discrete and noisy contacting force patterns in the visualization\ninterface. On the other hand, a more reasonable color bar setting can also enable some\nsmaller contact forces to be displayed as activated in the visualization interface.\nTherefore, as shown in Fig. R2-5(a) to Fig. R R2-5 (c), we retested the three actions of\nthe dumpling-making task: kneading, pressing, and pinching, especially showcasing the\nfirst two actions.\nFig. R2-5. A dumpling-making task and the results of tactile response and normalized pressure in\nthree actions: (A) kneading, (B) pressing, and (C) pinching.\nIn the kneading action, the position of the contact point between the dough and the\npalm area continuously scrolls clockwise, maintaining a high degree of consistency\nwith the real situation. In the pressing action, due to a significant downward pressure\ncaused by the entire palm, many sensing points in the palm area are illuminated. In the\nsecond visualization interface of Fig. R2-5(b), only the bottom area of the palm is used\nto apply pressure, which can also be reflected in the VR visualization interface. The\nvisualization of the pinching action has also been updated. In addition, Fig. R2-6 shows\na more continuous and smooth transition of the contacting force patterns in the kneading\naction, hoping that these results can help readers better understand the dynamic\noperation process. The above related images and videos have been updated in the\nresubmitted manuscript and supplementary files.\nFig. R2-6. The action of kneading and real-time VR interface showing the force distribution.\n5. Also, when using the fingertips to press the dough, the signals from the fingertip\nregions are not particularly obvious.\nAuthors:\nThank you for kindly reminding us. The reason why the tactile signal in the\nfingertip area is not particularly obvious when pressing the dough is due to the original\nvisualization approach. According to our original color bar setting, if the voltage caused\nby force is still less than 3 V, it will continue to display yellow, so no change is observed.\nWhen the user wears the tactile glove and keeps the same force as pressing the dough\nto touch the indenter shown in Fig. R2-7(a), a force curve can be obtained (Fig. R2-7\n(b)), of which the average force is less than 0.25 N. To solve the above problem, we\nconsider setting the color bar to a non-linear color change, so that it can be clearly\nobserved even in a small force range. Fig. R2-8 shows the change in force when five\nfingertips respectively touch the dough after using the new color bar. It can be seen that\nthere is a significant reddish color response in the contact area of the fingertips, which\nis a significant improvement over the original. Moreover, as shown in Fig. R2-9, the\nproposed tactile glove has a high sensitivity of 271.26 kPa-1 within the range of 100\nkPa, and thus is sufficient to detect small force changes in hand-object interaction tasks.\nThanks again for pointing out this issue!\nFig. R2-7. The contact force measurement of the fingertip by commercial test equipment and (b)\nthe sensing force curve\nFig. R2-8. The force signals when five fingertips press the dough, respectively\nFig. R2-9. The relative resistance variation of a tactile sensing unit in the range of 0 to 400 kPa\n6. There is a lack of detailed information regarding the frame rate used during the tactile\nglove’s operation, which is crucial for evaluating the glove's responsiveness and\napplicability in real-time applications.\nAuthors:\nWe appreciate your concern about the frame rate of the ViTaM system. The\nminimum frame rate of our proposed system for tactile perception is 13 Hz, which\nmeans a time delay of 77 ms. Fig. R2-10 shows the screenshots of the implementation\ntests.\nFig. R2-10. Screenshot of frame rate test\nA detailed elucidation of several significant sources of delay will be provided.\nSpecifically, the following points are addressed:\n(1) The scanning circuit proposed in this work is composed of a 24-row by 48-\ncolumn array, where each row is connected to the output of a decoder (74HC154,\nNexperia), and each column is connected to the input of an Analog-to-Digital Converter\n(ADC). A total of 6 ADC chips, each with 8 channels, are utilized in the system. During\na single sampling event, each ADC channel captures 2 bytes of data. Given that the\nADC operates at a frequency of 4 MHz, a single sampling event requires approximately\n4 μs. Furthermore, the establishment time must also be taken into account, hence, an 8-\nchannel ADC chip requires approximately 36 μs to sample eight data points. Under the\nassumption that the delay associated with the decoder’s row switching is neglected, the\nsampling of 1152 data across a 24-row by 48-column array would require a total of\napproximately 5.184 ms;\n(2) Due to the employment of serial port communication as the wired transmission\nmethod in this work, the latency associated with the serial port’s data transmission has\nbecome a predominant factor. A data packet in our study consists of 1152 data, with\neach data point occupying two bytes, totaling 2304 bytes. The baud rate of the serial\nport is set to 921600, which equates to 92160 bytes per second. Consequently, the time\nrequired for the transmission is calculated as 2304/92160 seconds, which equals 25 ms;\n(3) System call latency: When using epoll to retrieve data in power-saving mode,\nthe latency can reach up to 1 ms depending on the system scheduling;\n(4) Rendering latency: The VR visualization operates at 60 fps, which translates to\na time of 17 ms.\nIn summary, for daily operations such as hand grasping, a frame rate of 13 Hz is\nsufficient to meet the application requirements. In future work, the frame rate of the\ntactile glove can be improved through various means, such as selecting higher-\nperformance ADC chips, utilizing wireless transmission methods like WiFi or\nBluetooth, and operating the upper computer in a high-performance mode.\nThanks again for this question and we have added this content to the revised\nmanuscript.\n7. I noticed that many sensors have never lit up; have the authors checked whether there\nmight be any dead sensors?\nAuthors:\nThanks for pointing out this question. Before application, we tested all channels of\nthe glove demo, and only one channel did not work properly. The reviewer mentioned\nthat many sensors might never have been activated. This could be due to a lack of\napplied force during operation or because the color bar we set was not reasonable and\ndid not display the corresponding results. To address this issue, we applied normal force\nto different areas of the tactile glove with the newly set color bar to ensure that most of\nthe sensors could work normally, as shown in Fig. R2-11.\nFig. R2-11. The compression on the tactile sensing blocks to ensure the functionality\nFig. R2-12 and Fig. R2-13 respectively illustrate the force distribution on different\ntactile sensing blocks of the five fingers and the palm. For the palm area, the area\nindicated by the yellow box represents the tested region, while the area indicated by the\ngreen box denotes the region currently being tested. The number of normal sensing\nunits and the total number of sensing units in each area are displayed at the bottom of\neach figure. The proposed ViTaM system’s acquisition circuit comprises a total of 1152\nsensing channels. As an example, this VR interface integrates 456 channels, with 443\nchannels functioning correctly, resulting in a yield rate of 97.15%. The authors believe\nthat this yield rate is sufficient for human-machine interaction applications.\nAdditionally, due to the distance between adjacent sensing units being only 2.5 mm, for\nlocal bad points, adjacent points can be used as a preliminary replacement. If multiple\nconcentrated bad points appear, the modular design also facilitates local replacement,\nthereby meeting the performance requirements of tactile gloves.\nFig. R2-12. The illuminated sensing points of different tactile sensing blocks on the finger area\ncan work normally\nFig. R2-13. The illuminated sensing points of different tactile sensing blocks on the palm area can\nwork normally\nThe corresponding results have been added as supplementary information. Thank\nyou for your reminding!\n8. The noise from the sensor is also evident from Fig. 3. Characterizations of the tactile\narray. I: even when the strain is at 5%, the error rate is already 20%.\nAuthors:\nThank you for your question! We apologize for the negligence in our initial\nsubmission, where an incorrect set of curve data was included. Upon thorough re-\nexamination, we identified the error and have now updated Fig. 3I accordingly, as\nshown in Fig. R2-14. It shows the relative error between the estimated forces and the\napplied forces. Since the estimated forces without calibration are generally smaller than\nthe applied forces, the relative errors presented in the figure are consistently negative.\nAfter calibration, the estimated forces establish a strong consistency with the applied\nforces, proving the effectiveness of the proposed strain interference suppression. We\nregret any confusion this may have caused and appreciate your understanding and kind\nreminding.\nFig. R2-14. The relative error between the estimated forces and the applied forces under different\nstrains of 5%, 10%, 15%, 20%, 25%, 30%, and after corrected (abbreviated as crt)\n9. I am also curious about the interference from the hand pose. Will different hand poses\nalso lead to different tactile patterns and, if so, how to disentangle signals from the hand\npose and the actual physical interactions with the objects?\nAuthors:\nWe sincerely thank the reviewer for this professional question.\nFirst, we hold the opinion that the variations in physical quantities induced by hand\nposes primarily involve two types: strain and pressure. Strain, such as the bending\ncaused by hand extension or the stretching at interfaces when in contact with\ndeformable objects, represents the primary interference targeted for elimination in this\nstudy. This interference can be mitigated using the active suppression method proposed\nherein, which leverages symmetric response and adaptive calibration.\nSecond, the contact pressure resulting from hand poses, as you noted, indeed leads\nto different tactile patterns. However, it should be emphasized that the intensity of\npressure signals originating from the hand is significantly lower than that from\ninteractions with objects, with a difference typically exceeding tenfold. This will be\nverified in subsequent experiments, and such signals can be easily filtered out by a low-\npass filter. In terms of neural networks, we believe that they do not constitute significant\ninterference. Considering that as early as 2017, research (Int. J. Comput. Vision, 2020,\n128(7): 1867-1888) had demonstrated that even randomly initialized neural networks\npossess denoising capabilities. Under a supervised learning setting, signals related to\nthe supervised task (here, contact reconstruction) are enhanced, while unrelated signals\nare attenuated.\nTo elucidate the aforementioned issue, we supplemented our study with three\ngesture-based tasks that do not involve interaction with objects.\n(1) As shown in Fig. R2-15(A), the user wears the tactile glove and performs the\ndough-kneading gesture from the dumpling-making task without making contact with\nthe Table surface. The normalized pressure curve on the right indicates that the signals\ncollected by the glove are extremely weak, with amplitudes less than 0.1. We speculate\nthat these signals may result from minor hand movements. In contrast, Fig. R R2-15(B)\nillustrates the actual dough kneading action, with the amplitude of this curve being\napproximately 12 times greater than that in Fig. R R2-15(A). Thus, the two can be\ndistinctly differentiated.\nFig. R2-15. (A) The hand pose only task in the manipulation of kneading and (B) the actual dough\nkneading action with normalized pressure curves\n(2) In Fig. R2-16(A) and Fig. R2-16(B), we compare the manipulation of grabbing\nunder two conditions: hand pose only and interaction with a sponge. Initially, the hand\nis in an extended position, followed by five repetitions of a grabbing motion. The\nnormalized pressure curves are shown on the right. The figure highlights the noise\ninduced solely by hand pose during the five grabbing motions, with amplitudes\ngenerally below 0.05. In contrast, when the tactile glove interacts with the sponge, the\nnormalized pressure amplitude collected by the tactile glove reaches up to 0.8, which\nis 16 times greater than the noise induced by hand pose alone.\nFig. R2-16. (A) The hand pose only task in the manipulation of grabbing and (B) the actual\ngrabbing of a sponge with normalized pressure curves\nFig. R2-17. (A) The hand pose only task in the manipulation of pinching and (B) the actual\npinching of a paper cup with normalized pressure curves\n(3) Moreover, we performed a paper cup-pinching operation, with the results of the\nhand gesture alone and actual interaction shown in Fig. R2-17(A) and Fig. R2-17(B),\nrespectively. The paper cup-pinching motion was executed four times, and the\ncorresponding noise appeared on the normalized pressure curve. Due to the larger\namplitude in the third and fourth pinching motions, the normalized pressures were\nslightly higher than that of the first two, with amplitudes around 0.15. As a comparison,\nin the actual paper cup-pinching operation, since the cup contained water and each\npinching motion was maintained for 0.5 seconds, the normalized curve exhibited\namplitude changes greater than 0.9 along with a certain holding duration. The amplitude\ndifference between these two operations also reached 6 times.\nAdditionally, we are also inspired by the methodologies from the referenced\nliterature (Nature, 2019, 569(7758): 698-702): By subtracting the frame with the\nmaximum hand pose signal from the frame with the maximal object information, it can\ndecompose the tactile map into the hand pose signal and the object-related pressure\nmap. In future work, we will further explore methods to distinguish between hand poses\nand operations involving only minor normalized pressure variations to enhance the\nindependence of the collected pressure signals.\nThe corresponding results have been added as supplementary information. Thank\nyou once again for your insightful questions!\n10. “The repeatability of the tactile array under a load pressure of 5 kPa over 2000\ntimes,”—what about drifting after extensive use with human hands?\nAuthors:\nThank you for your pertinent question. In the original manuscript, we focused on\nevaluating the repeatability of the tactile array under a load pressure of 5 kPa over 2000\ncycles. We recognize that extensive use of human hands can introduce additional factors\nthat may affect performance, such as sweat, skin oils, and aging issues of piezoresistive\nmaterials and packaging.\nTo specifically address the concern of drifting after extensive use, we have initiated\nadditional tests by using a tactile glove manufactured a year ago, involving prolonged\nand varied interactions with human hands. Firstly, Fig. R2-18(A) illustrates the results\nof repeatedly applying a 5 kPa load pressure 100 times to a sensing module of the tactile\nglove after one year of use. It is evident that the relative change in resistance remains\nat 40%. Moreover, the insets in Fig. R2-18(A) reveal that the waveform during each\napplication and release of pressure exhibits a high degree of consistency; Second, the\nuser wore the tactile glove and repeated the action of grasping a sponge at a frequency\nof 2 Hz for 2000 cycles. The relative change in resistance was then measured in the\nsame manner as described above, with the waveform shown in Fig. R2-18(B). The\nresults indicate that the tactile sensing module still exhibits good consistency in its\nresponse to the same pressure; Finally, the user continued to wear the tactile glove and\nperformed various object manipulations continuously for 5 hours. Subsequently, using\nthe same testing method as described above, a 5 kPa pressure was applied 100 times.\nThe results, shown in Fig. R2-18(C), reveal that the maximum relative change in\nresistance decreased from 40% to approximately 38%. This reduction is likely due to\nthe aging and wear of the piezoresistive material.\nFig. R2-18. The relative resistance changes of the tactile glove when it was used (A) for one year;\n(B) after continuously conducting the action of grasping for 2000 Hz, and (C) for object\nmanipulations for 5 hours\nTo sum up, the overall tactile sensing performance remains acceptable and within\nfunctional limits. Our results indicate that while some degradation is present, it does\nnot significantly impair the tactile glove’s effectiveness for most practical applications.\nNecessary recalibration before long-term use is essential. In the future, we will continue\nto refine the calibration principles and calibration intervals for the tactile glove. The\ncorresponding results have been added as supplementary information.\nIn addition, there was a mistake in the y-axis labeling of the repeatability test in\nFig. 3K, which we have revised. We apologize for any inconvenience caused. Thank\nyou once again for your question!\n11. I’m also curious about the possibility of mass-producing the glove.\nAuthors:\nThank you for this comment. We are confident that this tactile glove holds great\npossibility to be mass-produced. The specific reasons are as follows:\n(1) The materials are readily available. First, multi-walled CNT aqueous dispersion\nand natural latex solution are required to fabricate piezoresistive films, which are very\ncommon and commercially available; Second, commercial conductive fabric wires\nserve as electrodes of the tactile sensing block, which makes it easy for the preparation\nof electrodes; Third, the silver paste and ultraviolet curing adhesive are also\ncommercially available. Therefore, there are sufficient sources of materials to support\nlarge-scale production.\n(2) The manufacturing process is simple and feasible. The steps to prepare tactile\nsensing blocks are provided in the METHODS section. It is estimated that preparing a\nbatch of tactile sensing blocks for 50 tactile gloves takes only 26 hours. As for the fully\nwoven wiring method, it can be completed using a digital embroidery machine, which\ncan meet the requirement of mass-producing. Additionally, the processing of FPC and\ncircuits is also based on mature technologies. The above process steps have been\nvalidated through small-scale processing.\n(3) The cost is manageable. In Supplementary Table 2 and Table 3, we mention that\nthe cost of manufacturing a tactile glove is $3.38, and the cost of manufacturing a set\nof hardware circuits is $26.63. This is lower than the costs reported in this literature\n(Nature 569, 698–702 (2019), DOI: 10.1038/s41586-019-1234-z). Furthermore, with\nlarge-scale production, there is potential for further price reduction.\nIn summary, the tactile gloves proposed in this manuscript have significant\npotential for large-scale production. The corresponding results have been added as\nsupplementary information. Thank you for your time!\n** Questions regarding the calibration process **\n12. Given the limited quality of the obtained tactile signal, a more comprehensive\ndesign and analysis of the calibration process is necessary.\nAuthors:\nWe appreciate your concern about the calibration process. Due to the susceptibility\nof the MWCNT-latex film developed in this work to strain interference, a closed-loop\nand adaptive calibration for quantitative detection of strain interference and force\ntesting results is urgently required. The following section provides a detailed\nexplanation of this closed-loop and adaptive calibration process:\nFig. R2-19. (A) (i) The relative resistive variations of the positive, and negative effect membranes\nand (ii) the output voltage with rise edges of the strain interference detection circuit. (B) The\nrelative resistance variation curves of the positive effect membranes and negative effect\nmembranes when applied with a strain increasing from 0 to 50%, which is named a symmetric\nresponse to the sensing error of strain. (C) Changes in the output voltage versus force curve of the\ntactile sensing unit under different strain interferences.\n(1) First, how can the presence of strain interference be determined? When the\nstacked positive and negative effect membranes are subjected to strain, a relative\nresistance change occurs, as shown in Fig. R2-19(A(i)), resulting in a corresponding\nrising edge in the output voltage of the strain interference detection circuit, as illustrated\nin Fig. R2-19(A(ii)). This rising edge in the output voltage serves as the flag for\nidentifying the presence of strain interference.\n(2) Second, how can the magnitude of the strain interference be quantitatively\ndetermined? From Fig. R2-19(A(i)), read the value of the relative resistance change\nrate on the Y-axis corresponding to the trough of the negative effect membrane (denoted\nas Y ). Since the positive effect membrane and the negative effect membrane exhibit\nx\nthe same amplitude of relative resistance change rate but in opposite trends when\nsubjected to the same strain, only one of the membranes needs to be selected for the\ndetermination. Based on Fig. R2-19(B) and Y , the magnitude of the strain interference\nx\nat that moment, denoted as ɛ , can be determined.\nx\n(3) Third, perform preliminary tests to measure the voltage changes of the\nstacked positive and negative effect membranes under several different strains.\nFor instance, in this work, force estimation curves were tested under seven strain\nconditions: 0%, 5%, 10%, 15%, 20%, 25%, and 30%, as shown in Fig. R2-19(C).\n(4) Fourth, how can the force estimation curve corresponding to the strain\ninterference ɛ be determined? In the previous step, we only tested force estimation\nx\ncurves under seven different strain conditions. Since it is impractical to enumerate force\nestimation curves for all possible strain interferences, this study proposes a method\ncalled “local domain curve interpolation” to update the force estimation curve\ncorresponding to ɛ . This interpolation process is implemented through Python.\nx\nAs shown in Fig. R2-20, we first identify the two values closest to ɛ among the\nx\nseven conditions (0%, 5%, 10%, 15%, 20%, 25%, and 30%). For example, if ɛ is 2.5%,\nx\nwe determine ɛ to be 10% and ɛ to be 0%. Next, based on Fig. R2-19(C), we\n_up _bottom\nidentify the force estimation curve corresponding to ɛ , denoted as curve (e.g., the\n_up _up\norange curve on the right side of Fig. R2-20), and the force estimation curve\ncorresponding to ɛ , denoted as curve (e.g., the blue curve on the right side\n_bottom _bottom\nof Fig. R2-20).\nThen, a new curve, curve (e.g., the green dashed line on the right side of Fig. R2-\n_x\n20), is interpolated between curve and curve in proportion to the distance\n_up _bottom\nbetween ɛ and ɛ , ɛ . This interpolated curve, curve , represents the force\nx _up _bottom _x\nestimation curve corresponding to the strain interference ɛ . Finally, using curve for\nx _x\nforce estimation, the force magnitude under the strain interference ɛ can be calculated.\nx\nFig. R2-21 illustrates the obtained interpolated force estimation curves when the strain\ninterferences are 2.5%, 5%, and 7.5%.\nFig. R2-20. The flow chart of the closed-loop and adaptive calibration process\nFig. R2-21. The interpolated curves under the strain interferences of (A) 2.5%, (B) 5%, and (C)\n7.5%\nIn summary, traditional tactile sensors have relied on a single, pre-determined force\nestimation curve for measuring force. The novelty of this work lies in dynamically\nadjusting the force estimation curve based on the magnitude of the strain interference.\nHowever, the authors also considered another issue: the force estimation curves in Fig.\nR2-19(C) may drift over time due to prolonged use of the tactile glove, a common aging\nphenomenon in piezoresistive materials based on conductive polymers. Therefore, it is\ncrucial to periodically calibrate the tactile glove.\nThe corresponding results have been added as supplementary information.\nThe authors claim that “The relationship between voltage change and loading force is\ndetermined through a mathematical fitting model.” A more detailed explanation is\nneeded.\n13. For example, how much data and what kind of data are used?\nAuthors:\nAppreciate this comment.\nFirstly, the tactile glove consists of 38 tactile sensing blocks, each sharing the same\nassembled composite film. During calibration, each tactile sensing block is tested three\ntimes, and the average of these tests is used to plot the relationship between the loading\nforce and voltage. When applying the loading force, the voltage is sampled every 0.5 N\nwithin the force range of 0-40 N, resulting in each force estimation curve comprising\n80 data points.\nFig. R2-22. (A) The proposed design of the pressure detection circuit of the tactile sensing blocks\nand (B) its simplified diagram of a single module\nSecondly, the data types used here are the loading force and the output voltage of\nthe tactile sensing blocks. The loading force is applied using a tensile tester (AGS-X\n50N, SHIMADZU) equipped with a customized probe. The voltage change of the\ntactile sensing blocks is collected through the circuit shown in Fig. R2-22(A). Fig. R2-\n22(B) provides the simplified diagram of a single module of the above circuit. The\ndesign of this circuit is described in the “Hardware implementation and simulation of\nthe scanning circuit” section in the Methods. Fig. R2-22 is added to the revised\nsupplementary information.\n14. Which mathematical fitting model is used, and how well does it generalize?\nAuthors:\nThank you for this question!\nWhen calibrating and fitting the force estimation curve of the tactile sensing block,\nwe found that the relationship between voltage and loading force generally adheres to\nthe following equation, namely the two-parameter Weibull distribution, often denoted\nas SWeibull2:\nV a(ab)e(k f)d (1)\nout\nwhere, f is the input variable of force, and V is the output variable of voltage.\nThe\nout\nparameters a and b are the two horizontal parameters of the function, representing the\nmaximum and minimum values, respectively. When the force is small, the voltage\napproaches b; as the force increases, the voltage gradually approaches a. Ideally, b is\nsupposed to be 2.5 V and a is supposed to be 5 V. The parameter k is a scaling parameter\nthat controls the stretch of the curve along the x-axis, while d is a shape parameter that\ncontrols the shape of the curve.\nBased on the calibration data for each tactile sensing block, our findings indicate\nthat the two-parameter Weibull distribution model provides an excellent fit for the data.\nDespite minor variations among different tactile sensing block samples, the overall\ntrend remains consistent. Fig. R2-23 depicts the relationship between output voltage\nand applied force for six samples positioned on the five fingers and the palm,\nshowcasing a high degree of consistency. The corresponding describe has been added\nas a supplementary note.\nFig. R2-23. The relationship curves between the output voltage and applied force of the tactile\nsensing blocks from the (A) thumb (FT1), (B) index finger (FT2), (C) middle finger (FT3), (D)\nring finger (FT4), (E) little finger (FT5), and (F) palm\n15. Are the generalization capabilities different for various regions of the hand and\nduring different interactions with the environment?\nAuthors:\nThank you for your insightful question. The generalization capabilities of our\ntactile glove indeed exhibit slight variation across different regions of the hand and\nduring different interactions with the environment.\n(1) Regional variability: Different regions of the hand, such as the fingertips and\npalm, have varying sensitivity levels and mechanical properties because the force\nestimation curves of different tactile sensing blocks are not the same, which can\ninfluence the tactile glove’s responses. The piezoresistive film utilized in this study is\na conductive polymer formed by doping MWCNTs into natural latex. Consequently,\nthe distribution of MWCNTs can impact the piezoresistive properties. However, as\ndemonstrated in our response to the previous question (refer to Fig. R2-23), different\ntactile sensing blocks exhibit consistent mathematical fitting models. We have\nconducted calibration tests of all tactile sensing blocks, and the data indicates that while\nthere are minor discrepancies, the overall trend remains consistent. The tactile sensing\nblocks with higher sensitivity are installed on the regions of fingertips, which are more\nfrequently involved in sensitive and detailed tasks. For regions subject to significant\nstrain, such as the palm or knuckles, we have selected tactile sensing blocks that are\nrelatively more resistant to strain interference for installation in these areas. Each tactile\nsensing block undergoes rigorous calibration prior to deployment, including sensitivity\ntesting and performance evaluation under varying strain conditions. Based on these\ncalibration results, the tactile sensing blocks are strategically allocated to appropriate\nregions to ensure optimal performance.\n(2) Interaction variability: The type of interaction with the environment also\nimpacts the tactile glove’s generalization capabilities. Interactions involving complex,\ndynamic forces at stretchable interfaces, such as manipulating soft or deformable\nobjects, introduce additional strain interference compared to more static interactions\nlike holding a rigid object. Compared to other piezoresistive tactile gloves, ours offers\nsuperior strain suppression capabilities in an active mode, including a symmetrical\nresponse to strain interference based on our proposed stacked positive effect membrane\nand negative effect membrane, and an adaptive calibration method for estimated forces.\nThe above strategy exhibits characteristics similar to advanced human tactile perception.\nFor instance, humans do not need to know the specific noise affecting their grip; instead,\nthey adaptively adjust the force of their grip based on the state of the object being\nhandled.\nWe are committed to further improving the generalization capabilities by refining\nour calibration algorithms and enhancing the tactile glove’s sensitivity and adaptability\nacross various regions and interactions. Your feedback is invaluable for guiding these\nfuture improvements. The corresponding results have been added as supplementary\ninformation. Thank you once again for your constructive feedback!\n16. What about the generalization capability when interacting with objects of different\nmaterials?\nAuthors:\nThank you for raising this important question. Our tactile glove design ensures that\ninteractions with objects of different materials do not significantly impact its\nperformance of tactile sensing and 3D reconstruction.\nFirstly, regardless of the type of solid material being interacted with, our primary\nfocus is on capturing the forces at the interaction interface. Thanks to the flexible and\nstretchable nature of the tactile sensing blocks in our proposed tactile glove, the glove\nexhibits excellent conformability during dynamic interactions with deformable objects\nsuch as plasticine or cardboard. This level of adaptability is unattainable with traditional\npurely flexible sensors (e.g., those based on polyimide substrates) or rigid tactile\nsensors (e.g., GelSight and other vision-based tactile sensors). Consequently, our tactile\nglove is suitable for interacting with a wide range of material types, including both rigid\nand deformable objects. This capability represents a significant advancement in human-\nmachine interaction tasks compared to traditional approaches.\nSecondly, the tactile sensing blocks in this study are made from natural latex, which\nprovides a high friction coefficient during object manipulation. This ensures a secure\ngrip, preventing slippage and consequently reducing interference factors when\nmeasuring forces on stretchable interfaces.\nThirdly, each tactile sensing block undergoes rigorous calibration, with pre-testing\nstrain ranges from 0 to 30%. We also employ interpolation fitting methods (refer to Fig.\n3H), ensuring that any strain interference within this range can be accurately calibrated.\nThis technique standardizes the tactile feedback across various materials.\nFourthly, during the training of our vision-tactile fusion deep learning framework,\nwe tested objects with varying hardness, texture, and elasticity. This ensured that the\nadaptive algorithms of the tactile glove could effectively compensate for these\nvariations. Fig. R2-24 illustrates the reconstructed objects with different materials,\ndemonstrating that the performance of the tactile glove remains stable across\ninteractions with various materials.\nFig. R2-24. The in-contact object reconstructions of eight objects with different materials\nAlthough our current results are promising, the range of materials tested remains\nlimited. We plan to further refine our adaptive algorithms and deep learning framework\nby incorporating additional prior knowledge to enhance the tactile glove’s performance\nacross a broader spectrum of materials. We appreciate your feedback and are committed\nto continually improving the glove’s generalization capabilities.\n** Questions regarding the joint learning framework **\n17. While the authors have conducted ablation studies comparing models taking both\nvision and touch and their counterparts that take either vision or touch as input, only\nthree qualitative examples are shown. For a more comprehensive understanding of the\nperformance, it is essential to demonstrate the benefit of the tactile glove through more\ncomprehensive qualitative and quantitative comparisons.\nAuthors:\nWe sincerely thank the reviewer for this insightful suggestion. I have noted that\nyour concern regarding qualitative and quantitative comparisons is consistent with the\ncomment raised by Reviewer #1. We have given this issue careful consideration and\nanalysis. Below is our detailed response to this concern:\nWe have compared the reconstruction performance with a different sensor, DIGIT,\nwhich is an optical vision-based tactile sensor requiring a 3D reconstruction step to\nobtain the deformation based on images. The DIGIT sensor is integrated into the system\n“VTacO” we have previously proposed1. The force reading of DIGIT is obtained by\ncalculating the linear relationship between the deformation and the force load based on\npre-calibrated material parameters. The tactile image has a higher resolution, typically\n512 × 512, but has a lot of redundancy and is non-trivial to obtain accurate forces since\nthe geometry estimation from the tactile image is still an open problem2,3.\nFig. R2-25. (A) The front view, back view, and side view of the optical sensors with poor\nwearability and conformality when they are fixed on the fingers; (B) Two types of hand poses\n(touching and grabbing) demonstrated by the above optical sensors\nIn contrast, our multiple-channel glove can directly read the force measurement\nwith an accuracy of 97.6%. Thus, the difference in data features will significantly\nchange the way we process the data. Since they contain different levels of information.\nThe glove-based tactile array and its system “ViTaM” proposed in this work are\ncompared with VTacO.\nFrom the output side, dynamic reconstruction of deformable objects has long been\ndiscussed in the computer vision community4,5. However, most previous works cannot\ngeneralize well to different deformable objects, not to mention compatible with\ndifferent object types (i.e. rigid, articulated, or deformable). Therefore, we believe our\ntask of dynamic reconstruction on different object types is novel in task settings.\nAnother factor to consider is the module arrangement in neural network design. To\nfacilitate the reconstruction in a dynamic setting, we adopt the cross-attention\nmechanism to fuse inter-frame features. It is also significantly different from the VTacO\ndesign.\nIn summary, the novelty of the reconstruction framework lies in two places: the\ntask setting, and the introduction of dynamic attention. Therefore, the comparison with\nthe baseline methods has also been discussed separately.\nSince the optical sensors are relatively thick and can’t reach high wearability and\nconformality (Fig. R2-25(A)), we wear them on the hands and demonstrate two types\nof hand poses for reconstruction: touching and grabbing (Fig. R2-25(B)). The\nquantitative results are shown in Table R2-1 and Table R2-2. The smaller the chamfer\ndistance, the better the performance. The last row of the table provides a comparison of\nthe performance between ViTaM and VTacO. If the difference between the two is less than\n0.1 cm, it is called “On par”, otherwise it is judged as “Better” or “Inferior”.\nTable R2-1:\nThe comparison between VTacO and ViTaM using touching poses.\nObj. Type Elastic Plastic Articulated Rigid\nChamfer Folding\nCup Bottle Sponge Plasticine Scissor\nDistance (cm) rack\nVTacO 0.937 0.945 0.546 2.314 1.002 1.534\nViTaM\n0.921 0.953 0.503 2.429 0.991 1.549\n(this work)\nPerformance\nBetter On par Better Inferior Better On par\nComparison\nWe can see from the two tables above that, although the two algorithms reach\nsimilar performances with hand-touching objects, ViTaM has outperformed VTacO in\nthe case of hand-grabbing objects. We figure that while gel-based sensors can obtain\nhigher-resolution local geometry information, such as sharp edges or severe\ndeformation, our new distributed design of the tactile glove can obtain more\ncomprehensive features when the occlusion is too severe for the vision information.\nSince hand-object interactions are more likely to involve complex manipulation of the\nobject, inevitably leading to more severe occlusions. Our ViTaM system and the\nproposed pipeline are a better choice for hand-object reconstruction. We also\ndemonstrate some qualitative results in Fig. R2-26.\nTable R2-2:\nThe comparison between VTacO and ViTaM using grabbing poses.\nObj. Type Elastic Plastic Articulated Rigid\nChamfer Folding\nCup Bottle Sponge Plasticine Scissor\nDistance (cm) rack\nVTacO 1.125 0.995 0.733 3.245 1.103 1.509\nViTaM\n0.954 0.985 0.467 2.106 0.925 1.501\n(this work)\nPerformance On On\nBetter Better Better Better\nComparison par par\nFig. R2-26. Qualitative results for comparing ViTaM-Glove and the DIGIT in the system of\nVTacO\nAnother aspect of the experiments we have done is to examine the effect brought\nby the temporal transformer module in the temporal feature fusion part. For testing the\nperformance without it, we only use a self-attention module to encode the fused feature\nand use the same WNF Predictor to decode the winding number values. The results are\nshown in Table R2-3.\nTable R2-3:\nThe error with or without temporal transformer (TT) module.\nObj. Type Elastic Plastic Articulated Rigid\nChamfer\nFolding\nDistance Cup Bottle Sponge Plasticine Scissor\nrack\n(cm)\nViTaM\n0.953 0.984 0.435 2.075 0.903 1.451\n(w TT)\nViTaM\n1.035 1.112 0.631 2.431 0.975 1.459\n(w/o TT)\nThe error shows significant effects of the introduction of the temporal transformer\nmodule, probably because of the consistency of the feature between the two consecutive\nframes in one video sequence, for the difference of forces in two frames indicate the\ndeformation of the object, or joint status changed.\nIn summary, the tactile glove and its ViTaM system proposed in this paper exhibit\nsignificant improvements over existing vision-based tactile sensors from both\nquantitative and qualitative perspectives. Its excellent wearability ensures superior\noperational flexibility in hand-object interaction tasks, and it also demonstrates obvious\nadvantages in reconstruction results. Thank you once again for your question!\nReferences:\n1. Xu, W. et al. Visual-tactile sensing for in-hand object reconstruction. in Proceedings of the 2023\nIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 8803–8812 (IEEE,\n2023).\n2. Yuan, W., Dong, S. & Adelson, E. Gelsight: high-resolution robot tactile sensors for estimating\ngeometry and force. Sensors 17, 2762 (2017).\n3. Wang, S. et al. GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a\nCompact Robot Finger. in Proceedings of the 2021 IEEE International Conference on Robotics\nand Automation (ICRA) 6468-6475 (IEEE, Xi’an, China, 2021).\n4. Li, K. et al. Chord: Category-level Hand-held Object Reconstruction via Shape Deformation. in\nProceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV) 9410-\n9420 (IEEE/CVF, Paris, France, 2023).\n5. Tsoli, A. & Argyros, A. A. Joint 3D Tracking of a Deformable Object in Interaction with a Hand.\nin Computer Vision – ECCV 2018 (eds. Ferrari, V., Hebert, M., Sminchisescu, C. & Weiss, Y.)\nvol. 11218, 504–520 (Springer International Publishing, Cham, 2018).\n18. Related to my previous point, it is also crucial to conduct a much more\ncomprehensive study on the model’s performance for the dynamic reconstruction of\nhand-object interactions.\nAuthors:\nThank you for your valuable feedback. Regarding dynamic reconstruction, in\nresponse to earlier comments, we have conducted relevant ablation experiments, such\nas adding and removing the temporal transformer module. These have demonstrated\nthe model’s effectiveness in perceiving information from previous and subsequent\nframes and have also improved the dynamic reconstruction.\nTable R2-4:\nThe per-frame and full-video reconstruction error.\nObj. Type Elastic Plastic Articulated Rigid\nChamfer\nFolding\nDistance Cup Bottle Sponge Plasticine Scissor\nrack\n(cm)\nPer-frame 0.953 0.984 0.435 2.075 0.903 1.451\nFull-Video 0.961 0.981 0.489 2.103 0.958 1.440\nFull-Video\n0.00449 0.00185 0.00960 0.103 0.0156 0.0202\nVariance\nMoving forward, we not only compare the error of each frame’s reconstruction, but\nwe will also present the average error across the entire sequence to assess the model’s\nperformance in dynamic reconstruction. The results are shown in Table R2-4. The “Per-\nframe” row reports the chamfer distance calculated the mean per-frame error for the\nentire category. While the “full-video” row reports the chamfer distance calculated by\nfirst average on a full video, then on a whole category. Since the difference in frames\nbetween different manipulation processes. These two metrics have different numbers.\nWe also report the variance in the full-video metric to show the reconstruction\nconsistency between the frames. Additionally, we tested the model’s average inference\nruntime on one Nvidia RTX 4090 GPU, which is around 3-5 frames per second.\nThe corresponding results have been added as supplementary information. Thank\nyou for this professional question!\n19. The authors claim that their framework enables “the tracking and geometric 3D\nreconstruction of manipulated arbitrary objects.” This significant claim requires\nsignificant evidence. The authors should be more concrete about what type of objects\ntheir framework can reconstruct (e.g., rigid, elastic, plastic, or articulated) and provide\nsufficient evidence to justify the claim.\nAuthors:\nThank you for your insightful suggestion. We have categorized the objects in our\nmanuscript into several types: paper cups, bottles, and sponges as elastic objects,\nplasticine as plastic objects, scissors as articulated objects, and folding racks as rigid\nobjects. We apologize for any confusion that our description may have caused. We will\nclarify the types of objects more explicitly in the resubmitted manuscript and specify\nthe category of each object in the quantitative analysis of our experimental results to\nfacilitate distinction.\n20. For the hand-object reconstruction, there appear to be a lot of gaps between the hand\nand the object, especially in the scissor example. Would priors like contact, force, or\nform closure be helpful here?\nAuthors:\nWe appreciate your valuable comment. Firstly, we would like to confirm if the\nreviewer is concerned about the gaps in the scissors shown in the blue circles in Fig.\nR2-27?\nIn fact, these seeming gaps between the hand and the object are due to the occlusion\nof the hand in the visual information. A deeper reason is that a thin structure does not\nhave too many points in the observed point clouds. What’s worse, the thin structure is\non the edge, which is known to scatter the structural light from the 3D camera and result\nin even fewer points. Thus, thin structures like the scissor handles are likely to have\nbreaks in vision-only reconstruction. On the other hand, the sensors on the glove that\nare in contact with the object are able to get information about the object’s interaction\nwith the hand, which is just enough to complement the vision-only occlusion problem.\nFig. R2-27. The in-contact object reconstructions for two elastic sponges and a rigid scissor\nwithout and with strain interference suppression\n21. It seems that the hand pose is estimated from the point cloud. Will the cables on the\nglove and the glove itself interfere with the pose estimation results? Meanwhile, is it\npossible for the glove to directly provide the ground truth hand pose, either through the\nobtained tactile signal or some other integrated sensors?\nAuthors:\nThank you for your insightful suggestion!\nAs you mentioned, the hand pose is estimated from point clouds captured by a\ndepth camera, and we believe that the effects of the cables on the glove and the glove\nitself on hand pose estimation can be ignored. Although the score of the real hardware\nis slightly lower than that on the simulation, the overall impact is not significant due to\nthe following three reasons: (1) When generating data on the simulation end, substantial\nGaussian noise has been added to the point clouds introduced into the network for\ntraining, thereby endowing the point clouds with a degree of generalizability; (2) Our\nsimulation-end data consists of point clouds from a single view, hence even in the\npresence of occlusions or noise, the network is capable of effectively reconstructing the\nshape of the hand; (3) Our network takes into account the input from both preceding\nand subsequent frames, thus incorporating prior knowledge from the previous frame for\nhand pose estimation. Consequently, the overall impact is minimal.\nRegarding the second issue, both approaches are promising to obtain ground truth\nhand poses, but with non-trivial additional efforts.\nFor the first suggested approach, we can match the observed hand-object geometry,\nand tactile signals with a pre-determined template. The observed point cloud will help\nto estimate both the hand and object pose, while the tactile signals will facilitate the\ncontact location and refine the pose. However, it can be effective if the object is rigid.\nWhen it comes to deformable objects, we have to deal with the deformation to get the\nlocations of the contacted fingers right.\nFor the second suggested approach, we can integrate additional sensors to the\ntactile glove, such as strain sensors, electromyography (EMG) sensors, or inertial\nmeasurement unit (IMU) for hand pose estimation. This is an effective strategy that\nintegrates various functionalities into a tactile glove, thereby allowing the completion\nof the entire process with a single hardware. For example, Servati et al. proposed a\nsmart glove with stretchable helical sensor yarns (HSYs), IMUs and stretchy\ninterconnects. Each glove is equipped with 25 HSY sensors, nine of which are capable\nof detecting pressure. These HSY sensors can detect localized deformations caused by\nthe movement of joints and fingers or interactions with objects, including both\nextension and compression forces. In contrast to our approach, the goal of this study is\nto achieve accurate tracking of dexterous hand movements during object interactions,\nwhich only requires the extraction of different hand gestures. Our work, however,\nfocuses on the high-density, distributed pressure measurement of the entire hand during\ncomplex manipulations, for the fine-grained reconstruction of hand poses and\ndeformable objects. Therefore, the hand movements involved are far more random than\nthe types of gestures, and there is a higher demand for the sensor density on the tactile\nglove. Table R2-5 compares several technologies that could potentially be integrated\nwith the tactile glove for the purpose of hand pose estimation.\nTable R2-5:\nComparison of main techniques for hand pose estimation.\nHand pose estimation Force sensing\nRef.\nChannel Recognition Channel\nSensors Sensors\nNumber Results Number\nHelical sensor yarns 50 dynamic\n[1] ( HSYs) gestures + 48 9\n+ 2 yarns\n+ 9-axis IMUs static gestures\nElectromyography\n[2] 64 21 gestures \\ \\\nsensors\n50 dynamic\n[3] Triboelectric sensors 15 \\ \\\ngestures\n[4] Strain sensors 5 11 gestures \\ \\\nFinger-joint\n[5] Capacitive sensor 44 \\ \\\nangles\nPoint Any hand Piezoresistive\nOurs Depth camera 1152\ncloud poses sensor\nIn the early stages of our research, we also considered the option of integrating\nadditional sensors into the glove. However, several issues also require attention. Firstly,\nwhile integrating multiple IMUs on the back of the finger joints could accurately obtain\nthe hand poses, this method inevitably leads to a bulky design of the glove.\nConsequently, the flexibility of operation is significantly reduced, which in turn limits\nthe application prospects of this work; Secondly, our research objective is the fine-\ngrained reconstruction of deformable objects during manipulation. Therefore, obtaining\nonly the hand poses is insufficient; the morphology of the deformable objects must also\nbe captured. If one were to rely solely on a high-density tactile glove to physically touch\nand model the object completely, it would consume a considerable amount of time, and\nthe efficiency would not meet the demands of practical applications. Therefore, we\ncontinued to employ visual strategy, which stands as the gold standard for object\ntracking, to acquire “global” information during manipulation, and supplement it with\nhigh-density tactile information where the field of view is obstructed, finally offering a\nmore efficient and fine-grained reconstruction result.\nWe also appreciate the suggestions from the reviewer, which have inspired us to\nexplore more innovative methods in the future for obtaining integrated techniques that\ncan capture both global and fine-grained morphologies. For instance, this could involve\nincorporating miniaturized cameras or ultrasonic sensors into the tactile glove.\nReferences:\n1. Moin, A. et al. A wearable biosensing system with in-sensor adaptive machine learning for hand\ngesture recognition. Nat. Electron. 4, 54–63 (2021).\n2. Tashakori, A. et al. Capturing complex hand movements and object interactions using machine\nlearning-powered stretchable smart textile gloves. Nat. Mach. Intell. 6, 106–118 (2024).\n3. Wen, F., Zhang, Z., He, T. & Lee, C. AI enabled sign language recognition and VR space\nbidirectional communication using triboelectric smart glove. Nat. Commun. 12, 5378 (2021).\n4. Zhou, Z. et al. Sign-to-speech translation using machine-learning-assisted stretchable sensor\narrays. Nat. Electron. 3, 571–578 (2020).\n5. Glauser, O., Wu, S., Panozzo, D., Hilliges, O. & Sorkine-Hornung, O. Interactive hand pose\nestimation using a stretch-sensing soft glove. ACM Trans. Graph. 38, 1–15 (2019).\n22. For the dynamic reconstruction, both hand and object are modeled by FEM, and the\nsimulation environment to generate training data is supported by RFUniverse. It would\nprovide readers with a more comprehensive understanding if the authors could provide\nmore details and visualizations of the simulation.\nAuthors:\nFor our simulation environment, it consists of two main parts: Modeling and\nRendering.\n(1) As for the modeling process, our environment is based on FEM and ABD (affine\nbody dynamics)1, and augmented by a precise contact model, IPC2. Specifically, we\nmodel the MANO hand as articulated bodies, while the manipulated object as a\ntetrahedral body. The rigidity and softness of the object can be adjusted by the physics\nparameter, Young’s Modulus, and Poisson’s ratio. The experiment workflow is as\nfollows:\na) We load the hand and the object models into a simulation, attach physics\nparameters, set up an initial configuration that is intersection-free, and perform\ngrasp planning with “GraspIt!”3.\nb) We control the hand to move towards the planned grasp pose via the auto-grasp\nfunction in “GraspIt!”. Providing target joint states, the simulator automatically\nupdates the scene according to physics-based dynamics and additionally\nprovides informative vertex-level contact force.\nc) We examine the contact force in every frame. Once we detect hand-object\ncontact, we tighten the grasp for another several frames and then terminate the\nsimulation.\nThroughout the simulation, we save scene meshes, body states, and contact forces\nfor every frame.\n(2) After modeling the interaction process, we obtain object and hand meshes, and\ncontact force for every vertex in each frame. We use Unity and RFUniverse to conduct\nthe rendering procedure, which consists of two functions for every frame:\na) Obtaining depth image and point cloud. Object and hand meshes undergo the\nfollowing transformations: image space -> camera space -> world space. In the\nimage space, the vertices include depth information relative to the camera. Then,\na custom shader is used to linearly map the depth within a certain range to\ngrayscale values and shade the pixels accordingly, which are eventually\nrendered onto the screen. The point cloud in the world space is calculated from\nthe depth image we obtained based on intrinsic parameters and the extrinsic\nmatrix of the cameras.\nb) Calculating the sensor readings. In Unity, we pre-set the sensor positions based\non an open hand. Since we have the force value of every vertex on the hand, we\ninterpolate the values to the exact sensor positions based on the hand joints’\nstates, and can thus obtain the sensor readings.\nFig. R2-28 shows the rendering process in Unity. We can see that for these three\nframes, the object has varying degrees of deformation. The depth images are captured\nand sensor readings are calculated.\nFig. R2-28. Simulation environment demonstration\nReferences:\n1. Lan, L. et al. Affine body dynamics: fast, stable and intersection-free simulation of stiff materials.\nACM Trans. Graph. 41, 1–14 (2022).\n2. Li, M. et al. Incremental potential contact: intersection-and inversion-free, large-deformation\ndynamics. ACM Trans. Graph. 39, 49 (2020).\n3. Miller, A.T. et al. Graspit! A versatile simulator for robotic grasping. IEEE Robot. Autom. Mag.\n11, 110-122 (2004).\n23. It would also be helpful to provide a quantitative measurement of how the\nsimulation matches the data collected in the real world, especially considering the noise\nfrom the sensors.\nAuthors:\nThank you for your valuable suggestion. In a glove-existing hand-object\nmanipulation setting, there are two types of contact namely object-glove contact, and\nglove-hand contact.\nFor the object-glove contact, the error is dependent on the glove calibration process\nfrom the real world, which was discussed earlier. In the simulation environment, the\ncontact status between hand and object is modeled by FEM and IPC, which can assure\nhighly accurate simulation of per-vertex force. We repeat the calibration action in the\nsimulator. As shown in Fig. R2-28, it gives us an error of the order of 0.05 N.\nFor the glove-hand contact, it can cause strain. Since in our simulator, we directly\nuse the MANO hand to contact with the object, thus such strain is not simulated in our\nsimulator. However, thanks to the strain suppression techniques proposed in our work,\nthe influence of the strain is minor.\nOn the other hand, for the visual input part, we used the Photoneo MotionCam M+\nmodel, with the error of the order of 5e-4 m.\nIn our implementation, we add Gaussian noise on the tactile and visual sensory\ninformation from the simulator with distribution N(0, 0.025N) and N(0, 0.00025m)\nrespectively. The variance is set to make sure 95.45% (2𝜎 confidence) of random\nnoises are in the range of measured error.\nReviewer #2 (Remarks on code availability):\nThe authors have included detailed instructions on how to run the application in the\nrepository, but I didn't get the chance to install and run it myself.\nAuthors:\nThank you for your comment. We have updated the code repository for the\nalgorithm part and provided a small sample for you to run and test. The repository link\nstays the same: https://github.com/jeffsonyu/ViTaM. We apologize for any\ninconvenience caused previously."}, {"round": 2, "reviewer_comments": {"reviewer_1": "I have reviewed the revision and the rebuttal. I thank the authors for their efforts and agree that my concerns have been addressed. I thus recommend acceptance.\n--- Original Review ---\nIn this article, the authors proposed a visual-tactile recording and tracking system (ViTaM), which includes a stretchable tactile glove and a visual-tactile joint learning framework to estimate dynamic hand-object states during manipulation. The authors validated their findings by computing the accuracy in force measurement and the shape reconstruction error for both deformable and rigid objects.\nAs my expertise is in artificial intelligence, not hardware design, I'll primarily focus on the learning framework. I think the proposed learning framework is not novel, and surprisingly, comparisons with existing methods are lacking. In this case, it becomes impossible to assess whether the proposed framework has an advantage, either scientifically or empirically, over prior art in this area.\nSpecifically, in the supplementary material, the authors summarized a number of existing methods that perform visual-tactile reconstruction of object geometry. These methods mostly build on publicly available sensors, such as GelSight and DIGIT. Thus, it seems a natural and required step to replicate and compare with some of these systems (with their released code) in order to demonstrate the superiority of the proposed shape reconstruction methods.\nWithout such comparison, we may only assess the reconstruction quality based on qualitative results. In addition to methods that take multiple sensory inputs, there has been fast progress in vision-based shape reconstruction of hand-object interaction [1, 2, 3], including some from the authors' institution [2] or from the authors themselves [3]. The authors should also compare the proposed system with these methods and explain the superiority of the proposed joint learning framework.\n[1] Chen et al. Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild. AAAI 2023\n[2] Li et al. CHORD: Category-level Hand-held Object Reconstruction via Shape Deformation. ICCV 2023\n[3] Xu et al. Visual-Tactile Sensing for In-Hand Object Reconstruction. CVPR 2023\nThe submission's presentation is fine. I have two suggestions: the first is that there are places where it feels unclear whether ViTaM refers to the learning framework only or the entire system (hardware and software together). To avoid confusion, it would be beneficial to clarify whether ViTaM is a comprehensive system that includes both the learning framework and the hardware components.\nThe second and more important issue is that the introduction is written in a confusing way. It spends much space on the challenges in hardware design without proposing a solution. When it finally introduced ViTaM as a solution to these challenges, it became hard to understand the motivation behind all design choices and how they addressed the challenges listed in the very long, definitely-too-long second paragraph of the introduction. I believe the introduction would benefit from a rewrite.", "reviewer_2": "I appreciate the authors' thorough and detailed responses to my questions. The additional analysis, clarifications, and new visualizations offer valuable insights into the glove's functionality, effectively helping readers understand the overall performance of the hardware and learning system, as well as its advantages and limitations.\nAlthough I still have some concerns about noise in the sensing results and would like to see reconstruction on a broader range of objects, I believe this paper provides unique value to the community through its applications in modeling hand-object interactions."}, "author_response": ""}]

Decision letter