Flexible robotic hand harnesses large deformations for full-coverage human-like multimodal haptic perception

Domain: physics
Published: 2025-12-14
Editorial outcome: minor_revision

Abstract
Integrating human-level haptic perception into soft grippers promises safer robotic grasping and improved human–robot interaction. While visual-tactile sensors promise high perception resolution at low cost, they often sacrifice compliance to maintain optical stability, hindering non-planar contact perception. We present FlexiRay, a soft gripper that integrates visual-tactile sensing with the Fin Ray Effect to achieve high compliance, broad sensory coverage, and multimodal capability. Combining a multi-layered flexible substrate, an optimized multi-mirror optical system, and a decoupled deep learning framework, FlexiRay replicates five of the seven human tactile modalities, including force, contact location, texture, temperature, and proprioception, with a single camera. It achieves 0.17 N force accuracy, 0.96 mm spatial resolution, 0.24 mm proprioception accuracy, and 1.17 °C temperature accuracy while maintaining over 90% effective coverage. FlexiRay empowers compliant grasping, safe collaboration, and intelligent teleoperation, underscoring its potential to propel service robotics toward enhanced intelligence, safety, and real-world utility.

Introduction
Humans sense of touch represents one of the most sophisticated perceptual mechanisms, comprising three core systems: the cutaneous/tactile sensing, the kinesthetic sensing, and the central nervous system1–4. These highly-interconnected systems underpin the fine and rich haptic perception capabilities of the human hand, which encompasses seven key modalities, namely pressure, contact localization, texture, temperature, proprioception, vibration, and pain2, as seen in Fig. 1a. Together, these modalities empower humans to perform various intricate and dexterous tasks with high precision5. Translating this exceptional sensory ability to robots brings about significant benefits, yet significant challenges2,6,7. Over the past three decades, researchers have explored nearly all forms of sensing, such as resistive8,9, magnetic10, pressure-sensitive11,12, capacitive9, waveguide-based13,14, acoustic15, and thermal sensing16,17. A special focus has been placed on the development of tactile sensing in soft robotic grippers13,15,18, aiming to enable more dexterous and safer environmental interactions via the combination of structural adaptivity and tactile perception. Despite substantial advancements, achieving large-area, high-resolution, and multimodal sensing remains a tremendous challenge19–21. This can be primarily ascribed to two factors: the high production cost of taxel-based measurement methods8 and the limitations in spatial resolution of data-driven computational sensors14–16,22. Therefore, existing designs often struggle with the trade-off among resolution, coverage, cost-effectiveness, and multimodal sensing capabilities. A comprehensive, human-like haptic perception solution remains an elusive goal.

Visual-tactile sensors (VTS) have emerged as a promising solution to address these challenges by leveraging metal-oxide-semiconductor optical arrays to convert multimodal touch information into high-resolution, pixel-level images17,23–26, thereby enabling insights into interactions such as pressure23,27, texture28,29 and proprioception30–32. Most existing VTS systems require stable optical paths to avoid occlusion and perception disturbances, which results in bulky designs and a heavy reliance on rigid structures18,28,33–39. This rigidity leads to a fixed sensing coverage, presenting significant challenges for integrating VTS with flexible grippers, as it conflicts with the compliance and flexibility inherent to soft robotic systems. Although some studies have achieved integrated robotic finger designs through optical path optimization37,40 or camera arrangement adjustments34,35,40, these grippers lack the structural compliance necessary for safe interaction. In particular, their structural rigidity restricts perception to planar contact areas, yielding only local, discrete information on non-planar or irregular objects, thereby reducing their versatility and adaptability in complex environments41,42.

To enhance perceptual capabilities during interaction, previous research has explored ways to improve structural compliance while preserving perceptual coverage. For instance, Insight19 and Svelte40 have attempted to introduce a hollow skeletal framework and a curved mirror back plane with material-based elasticity, respectively, yet they allow only limited deformation. Multi-segment fingers with dedicated cameras balanced dexterity and perception6,30, though each phalanx remains fundamentally rigid. Likewise, systems like JamTac43 achieve high compliance through jamming, but this mechanism inherently restricts their grasp range and load capacity, narrowing their practical utility. In comparison, bio-inspired Fin Ray Effect (FRE) grippers enable adaptive and stable grasping via passive structural deformation44. The integration of VTS with compliant FRE architectures offers a promising direction to mitigate the stiffness typical of traditional VTS45–47. Recent advances in this area include replacing rigid acrylic with thin Mylar films36,37, as well as monolithic silicone molding designs31 and spatial polyhedral networks41, all achieving improved structural compliance. However, these designs still struggle with visual occlusion and loss of perceptual field under large deformations. To preserve optical functionality, many existing designs deliberately stiffen flexible structures, sacrificing the inherent softness of FRE structures. As a result, embedding VTS within highly compliant structures, without compromising high-resolution and large-area perception capabilities under large structural deformation, remains an unresolved challenge.

We present FlexiRay, a sensorized soft gripper that replicates five of the seven essential haptic perceptions in the human sense of touch. It passively adapts to objects, stably capturing fine contact textures while simultaneously perceiving force, contact location, proprioception, and temperature, even under substantial structural deformation (Fig. 1b). This synergistic design, incorporating the flexible FRE structure, VTS, and a multi-mirror array, not only achieves a sensing area of up to 1560 mm² but also ensures stable perceptual coverage (87.2% on average) across varying deformation states. Considering the non-negligible inter-modality coupling, we design decoupled multimodal networks to achieve multimodal perception with a low end-to-end latency of ~25.8 ms for force, contact localization, and proprioception tasks, and 111 ms for temperature sensing. The system delivers sensing resolutions of 0.17 N for force, 0.96 mm for contact localization, 0.24 mm for proprioception, and 1.17 °C for temperature estimation. In terms of mechanical performance, FlexiRay exhibits over 400% higher deformation capacity than comparable flexible VTS designs, along with excellent durability exceeding 40,000 load cycles. A detailed comparison between the representative state-of-the-art VTS sensors and FlexiRay is summarized in the Supplementary Table 1.

Results
Working principles of FlexiRay
The design of the proposed FlexiRay is illustrated in Fig. 2a. The system is primarily composed of a compliant finger framework, an optical system, and a multi-layered tactile sensing pad. The compliant finger framework includes a back beam, a front beam, and crossbeams. The back and front beams, made of thermoplastic polyurethane (TPU) material, provide elasticity similar to the tendons in the human hand, maintaining structural stability while transmitting forces and deformations. The rigid crossbeams act as the bones of the hand, being hinged to the ends of the back and front beams to provide movable joints. The optical system includes a camera for image capture, multi-segment reflective mirrors for enhancing the camera’s field of view (FOV), and flexible LED light strips for illumination. The tactile sensing pad consists of a PDMS-based substrate, an elastic silicone layer, a reflective layer, a temperature-sensing layer, and a light-shielding layer, as shown in Fig. 2b. The low-hardness transparent silicone elastic layer is used to enable texture mapping (Fig. 2c), while the PDMS substrate provides flexible support without compromising compliance (Fig. 2d). The external silicone reflective layer, blended with aluminum powders, enables fine contact texture recognition. Atop this reflective layer is the temperature-sensitive layer embedded with thermochromic materials, whose thermally induced changes in the markers can be captured by the camera. The outermost light-shielding layer is responsible for mitigating external illumination interference. Thus, FlexiRay exhibits sensitivity to external physical contacts and temperature stimuli, similar to the mechanoreceptors and thermoreceptors in human skin.

The key challenge in integrating VTS with flexible structures lies in the significant deformation of the soft robotic gripper during the interaction, which obstructs the camera’s FOV. Rather than limiting the flexibility of the hand, we address this issue through systematic optimization, incorporating a multi-mirror configuration to transform the negative structural deformation into a functional advantage. As shown in Fig. 2e, each mirror is independently attached to the back beam, passively altering the direction of the camera light path. This allows the discrete capture of perception regions that the camera’s FOV does not cover. Combining the views from multiple mirrors achieves continuous coverage of a large perception area under dynamic loading conditions. To ensure optimal compliance adaptation, the optical system layout is modeled as a 2D geometric parameter optimization problem in the lateral cross-section. The optimization objective is designed to maximize the coverage of the sensing regions captured by direct camera views and mirror reflections. The optimization parameters include the camera’s position along the bottom beam baseline, its shooting angle, the length of each mirror, as well as the distance and angle of each mirror relative to the back beam. CMA-ES is used as the solving tool. Deformation data for the back and front beam joint nodes under different loads are collected to serve as the prior information. This optimization process essentially aims to find the layout parameters that maximize the camera’s FOV coverage across various deformations, leveraging the passive FOV reconstruction capability of multiple mirrors to enhance the camera’s perception of blind regions in deformed views. The convergence curve of the coverage loss during the optimization process is shown in Fig. 2f.

Multimodal perception model construction and performance evaluation
To achieve comprehensive environmental perception, we construct a coordinated sensing architecture that integrates five perceptual modalities: force, contact position, proprioception, temperature, and texture recognition. This architecture adopts a modular design, where each modality is handled by a specialized model, and cross-modal information fusion enables effective solutions to specific high-level tasks, such as classification. As the front-end of all vision-based haptic perception tasks, a lightweight semantic segmentation model is developed based on the PP-LiteSeg framework (Fig. 3a). It accurately partitions the raw haptic image into four functional regions: the front beam skeleton, the front beam interaction region (direct perception), the mirror region (reflective perception), and the local contact region. This preprocessing step effectively removes environmental interference and provides clean inputs for subsequent perception tasks. Next, a proprioception model based on a ResNet50 backbone is constructed (Fig. 3b), taking the segmented front beam skeleton and the sensing region as input. This model estimates the normal contact force, the 3D position of the contact point relative to its reference frame, and the crossbeam node positions in the proprioceptive coordinates, using dedicated heads for force, position, and node estimation. Furthermore, a temperature regression model built on a U-Net architecture is developed (Fig. 3c). Using marker images extracted by a YOLO-based detector, the model captures spatial color variations induced by temperature changes and outputs a continuous temperature field. Combined with the local contact-region mask, it further enables numerical perception of contact temperature. Additionally, FlexiRay’s multimodal perception enables high-level tasks, such as fine texture recognition. A texture recognition model, built on a ResNet50 backbone (Fig. 3d), takes sensing region images as input and classifies contact textures via a classification head. This modular multimodal perception architecture effectively addresses the spatial deformation coupling between the flexible structure and the contact interface. Employing specialized models for each modality enhances the precision, robustness, and adaptability of perception. The coordinated operation of all five modalities provides FlexiRay with comprehensive environmental awareness, enabling it to perform complex manipulation tasks in dynamic and uncertain environments.

Figure 3g illustrates the estimated normal contact forces across 5000 distinct contact locations and loading conditions. The horizontal axis represents ground-truth measurements, while the vertical axis shows model predictions. The predicted forces achieve a root-mean-square error (RMSE) of 0.17 N and a correlation coefficient of 0.995, indicating near-perfect agreement with physical measurements and underscoring the high fidelity of the force-sensing model. Figure 3h presents the distribution of absolute errors in estimated 3D contact positions under the same 5000 loading conditions. The closer a point lies to the origin, the smaller the error. Statistical analysis yields a mean error of 0.96 mm with a standard deviation of 0.45 mm, confirming both the accuracy and stability of the model in spatial contact localization. For proprioceptive representation of finger configuration, Fig. 3i shows boxplots of localization errors for 14 sensing nodes on each side of FlexiRay under the same loading conditions. The right-side nodes exhibit a mean localization error of 0.21 mm (mean s.d.: 0.13 mm), while the left-side nodes yield 0.26 mm (mean s.d.: 0.15 mm). These results demonstrate consistently high spatial precision and robustness. To further evaluate proprioceptive accuracy during continuous interaction, we conduct tests where contact depth is incrementally increased from a randomized initial position to a target load. Figure 3j compares measured and estimated values of both normal force and contact depth over 10 repeated randomized trials. The model maintains high prediction fidelity and stability even under dynamic, continuous loading, validating its robustness in real-time proprioceptive sensing.

Figure 3e illustrates the overall image-processing pipeline. The camera captures images at 30 fps (33 ms per frame), followed by image segmentation (18 ms) and proprioceptive inference, including force estimation, contact localization, and proprioceptive computation (78 ms). The end-to-end latency of the proprioceptive sensing task is ~25.8 ms. For the temperature-sensing channel, the core steps include marker extraction (29 ms) and temperature prediction (82 ms), resulting in a total latency of about 111 ms. Figure 3f presents the latency analysis and synchronization scheme across modalities. All sensing modalities share the same image stream, ensuring temporal consistency at the camera’s acquisition rate. The segmentation and proprioceptive perception modules operate at the same frame rate as image capture, while the temperature perception module processes one frame out of every four. This configuration achieves synchronized, low-latency multimodal perception while maintaining computational efficiency.

Exquisite texture detection and perceptual coverage
To assess FlexiRay’s texture detection performance, a series of test objects is selected, encompassing large curved or large-area items (e.g., screwdrivers, scissors), and curved objects of varying sizes and shapes (e.g., solder, bottle caps, vases, pen holders, clips, mice). A two-fingered gripper composed of FlexiRay is mounted on a UR5e robotic arm to perform natural grasping experiments and capture texture images without external interference. Several typical demonstrations are shown in Fig. 4a, with the complete set provided in the Supplementary Fig. 4. The results demonstrate that FlexiRay not only conforms seamlessly to and wraps around large curved objects but also accurately captures the surface contours and geometric details during flexible deformations. This showcases the extensive multimodal haptic perception capability of FlexiRay, which surpasses current VTS technologies.

The practical effect of the multi-mirror configuration on FOV extension is evaluated using the 3D-printed text ring shown in Fig. 4b. The ring has an outer diameter of 68 mm, a font height of 1.5 mm, and a linewidth of 1 mm. As shown in the internal view in Fig. 4c, under large deformations, the front beam obstructs the camera’s capture of the fingertip region. However, by incorporating mirror-reflective areas, FlexiRay is able to capture the occluded textures, thus providing a more complete perceptual field. Specifically, the direct perception area captures “RASPLA”, while the reflective perception area captures “ZJUGR”, forming a continuous texture pattern “ZJUGRASPLA”.

The perceptual coverage under different deformations is further quantified through both simulation and physical experiments. The simulation is conducted based on the collected hand configuration data at 200 random contact positions and loads. The camera’s FOV is uniformly discretized into 300 rays in a 2D plane, and the coverage of each ray on the contact area is calculated, including both direct incidence and mirror-reflected coverage. The contact area is discretized into 100 target points. Considering the varying propagation distance lc of each ray, the radius of a single ray is defined as R=lc 2(1−cosΔθ), where Δθ is the angle between two adjacent rays. The perceptual coverage is assessed by calculating the proportion of target points covered by each ray. Experimental results (Fig. 4d) show 87.2% average contact area coverage across all 200 random tests, with most of the coverage concentrated around 90%. Even under extreme deformations, the perceptual coverage remains above 70.0%. In particular, physical experiments further evaluate practical perceptual coverage stability under varying structural deformation. Rings with diameters ranging from 30 to 80 mm are chosen to mimic most real-world contact scenarios. The continuous alphabetic markers on the perceptual area help visually track coverage variations as the deformation changes. Results for the smallest (30 mm) and largest (80 mm) diameters are shown in Fig. 4e, with remaining cases provided in the Supplementary Fig. 6. Throughout all compliant grasping tests, FlexiRay consistently yields complete alphabet sequences, confirming near-full contact-surface coverage even across different deformation degrees.

Texture-based object classification during compliant grasping
FlexiRay’s large-area perception and fine texture sensing capabilities also facilitate more efficient and robust object classification during compliant grasping. This is evaluated by using eight textured sensory balls (diameters: 61–82 mm) as classification targets and comparing them with the state-of-the-art commercial VTS, GelSight Mini, which serves as the baseline. For each target, 80 haptic images are randomly acquired during stable grasps, yielding a total of 640 samples. The dataset is split into training and validation sets at a 4:1 ratio, and five-fold cross-validation is performed to mitigate potential distribution bias. The GelSight Mini sensor is employed under identical sampling, training, and parameter settings. As shown in Fig. 5c, FlexiRay achieves an average validation accuracy of 96.2% across folds, outperforming GelSight Mini’s 88.3% accuracy. The narrow variance across folds for both sensors indicates balanced dataset distribution. To highlight feature separability, Fig. 5b presents t-distributed stochastic neighbor embedding (t-SNE) visualizations of features from the best-performing FlexiRay model, extracted sequentially from the max-pooling, average-pooling, and fully connected layers, demonstrating both effective model design and training efficacy.

For independent evaluation, each gripper grasps every ball 15 times, generating two test sets of 120 samples each. As shown in the confusion matrices (Fig. 5d), FlexiRay achieves an average classification accuracy of 96.7%, with perfect accuracy for most balls and the lowest (86.7%) for the magenta ball, which is occasionally misclassified as cyan or purple due to similar edge textures. In contrast, GelSight Mini achieves an average of 85.8%, with markedly lower accuracy for magenta, purple, and cyan balls (66.7%, 66.7%, and 73.3%, respectively). These results underscore FlexiRay’s advantage: its sensing area (1560 mm²) is 5.9× larger than GelSight Mini’s (265.98 mm²), and its compliance enables single-contact conformation to curved surfaces, capturing non-planar features at greater distances. This capability even allows simultaneous detection of the magenta ball’s dual protrusions, which is impractical for GelSight Mini, leading to richer touch cues and superior recognition performance.

Using gradient-weighted class activation mapping (Grad-CAM) visualizations (Fig. 5e), we identify the regions most influential in classification. For instance, Magenta Test 1 leverages direct illumination over contact regions, whereas Magenta Test 2 focuses on specular reflection zones. Similarly, Cyan Test 1 relies primarily on direct-illumination cues, while Cyan Test 2 incorporates both reflection-dominant and direct-contact features. These examples highlight the importance of specular reflection sensing in soft visuotactile perception, enabling activation of distinct decision regions under varying contact configurations.

Building on the trained model, we implement a robotic ball-packing task (Fig. 5f). Balls are randomly mixed and dispensed onto an inclined feed channel, from which the robot grasps them sequentially. Using only tactile modality, the robot identifies surface textures and sorts balls into designated packing zones. Representative haptic images captured during sorting are shown in Fig. 5g. The task demonstrates FlexiRay’s compliance, grasp stability, and robustness in handling varied complex surface geometries, while relying exclusively on tactile sensing for accurate classification, underscoring its role in precise grasping and manipulation.

Compliant grasping, safe contact response and force-position sensing for interactive robot control
Grasping deformable or fragile objects requires not only structural compliance but also precise force-sensing capabilities. As illustrated in Fig. 6a and Supplementary Movie 3, FlexiRay, mounted on a UR5e robotic arm, sequentially grasps four soft and fragile objects from a table. Manually defined force thresholds are ~0.46 N for the bread (~20 g), 0.93 N for the chip (2.3 g), 1.12 N for the egg (44 g), and 1.88 N for the tape (25 g). These results demonstrate that FlexiRay achieves robust adaptability to delicate objects, ensuring grasping safety and stability through force closed-loop control, a capability critical for reliable operation in unstructured real-world grasping scenarios.

The capability to provide subtle perception, coupled with inherent compliance, can be further leveraged in safer and more versatile human–robot interaction tasks. Figure 6b illustrates the actuation and control system for both the gripper’s responsive evasion upon subtle contact and safe human-guided robotic arm manipulation, respectively. As shown in Fig. 6c, d and Supplementary Movie 4, upon detecting contact forces, the admittance controller generates compliant, force-adaptive motions based on pre-designed spring-damper characteristics, thereby ensuring interaction safety. However, it should be noted that large-area, low-texture objects (e.g., feathers) with minimal interaction forces pose challenges for conventional rigid vision-based tactile sensors, which rely heavily on the variation of surface normals. In contrast, FlexiRay achieves fine-grained sensing by leveraging structural deformation in response to external loads and integrating proprioceptive information with contact-region data. Figure 6c, d present the real-time force predictions collected during the experiments.

To further visualize the proposed model’s response mechanism across diverse contact scenarios, we employ the Structural Similarity Index (SSIM) and dense optical flow techniques to quantify continuous deformations and displacement directions of objects in sequential haptic images. As shown in Fig. 6d, the multi-mirror array on the dorsal contact surface, combined with proprioception, amplifies subtle deformations during large-area contacts, ensuring that FlexiRay stably delivers accurate tactile sensing under these challenging conditions. FlexiRay further demonstrates its potential as a human–robot interface. We show that a single FlexiRay can detect and control three DoFs of a UR5e robotic arm based on its perception of contact position and force. In contrast to previous setups, the spring property of the admittance controller is set to 0. The initial contact position is defined as the interface origin, with the currently detected contact position and force used to control the UR5e’s x, y, and z DoFs in the world coordinate system (see Fig. 6e and Supplementary Movie 4).

Multimodal haptic perception-enhanced intuitive physical human–robot interaction
Continuous contact localization during dynamic grasping is critical for stable grasping and human–robot interaction applications. To rigorously assess the contact localization capability of FlexiRay during dynamic interactions, we design a series of pen-tracing experiments using a high-precision visual motion-capture system (NOKOV, calibrated positional accuracy <0.8 mm) as the ground-truth reference. In four independent trials, a human operator holds a pen-nib and inscribes distinct trajectories (the letters “Z”, “J”, and “U” as well as the numeral “8”) directly onto FlexiRay’s soft visuotactile sensing pad. Throughout these free-form motions, the contact localization model continuously estimates the interaction point in real time.

As shown in Fig. 7a, the writing of “Z” serves as a representative example, displaying the three-dimensional pen-tip trajectory alongside raw haptic images captured by the internal camera. This visualization captures not only the spatial dynamics of contact, but also the evolving local deformation and texture cues encoded during writing. Quantitative comparisons between predicted and ground-truth trajectories (Fig. 7b, c) reveal consistently high spatial fidelity: mean localization errors are 2.68 mm (standard deviation, s.d.: 1.15 mm) for the “8” trajectory, 2.73 mm (s.d.: 1.04 mm) for “U”, 2.58 mm (s.d.: 1.13 mm) for “Z” and 1.93 mm (s.d.: 0.94 mm) for “J”. The relatively low s.d. values across trials confirm stable performance, even with variations in curvature, stroke direction, and contact dynamics. These results demonstrate that FlexiRay maintains sub-3-mm accuracy during non-repetitive, dynamic, multi-directional contact events, including under complex shear forces, structural compliance, and local material elasticity. The precision and robustness exhibited in this pen-tracing task highlight FlexiRay’s strong potential as an intuitive human–machine interface, capable of translating fine-grained contact position estimates into reliable control commands for robotic manipulation.

Global contact contour reconstruction
FlexiRay’s spatial proprioceptive perception further enables object shape reconstruction during dynamic grasping with improved efficiency. Two targets with distinct geometric characteristics are used, including a conical vase (truncated-cone lateral surface) and a French Classic Yellow Mustard container (low-curvature convex front surface, gourd-shaped side profile). Leveraging the high compliance, large contact coverage, and precise proprioceptive sensing, FlexiRay reconstructs contact contours efficiently and accurately with minimal trials (Supplementary Movie 6).

For the vase, FlexiRay performs a series of adaptive grasps at evenly spaced heights from base to top, where the inner camera captures high-resolution haptic images at each stable contact (Fig. 8a). With the proposed proprioceptive model, the spatial positions of key nodes on both fingers are estimated and mapped contact regions to a 3D coordinate frame. Spline interpolation is then applied to generate a continuous skeletal curve, followed by surface interpolation to reconstruct the sensing pads’ contact geometry (Fig. 8b). Contact regions extracted from the reconstructed surfaces reveal distinct local geometric features of the vase. Eight grasp heights are recorded (Fig. 8c), integrating local information from different contact regions, which yields a coherent analysis of the vase’s overall profile (Fig. 8d). Similarly, FlexiRay deploys adaptive grasps for the mustard container but across two distinct orientations. Seven positions on the front (Fig. 8h) capture the low-curvature near-planar geometry (Fig. 8f), while three lateral positions, aligned with different widths along the gourd-shaped profile (Fig. 8j), reveal its varying curvature (Fig. 8g). Local contact geometries from both orientations (Fig. 8i, k) are combined to further infer the object’s complete 3D contour.

These results demonstrate that FlexiRay efficiently acquires and integrates local shape features across multiple contact locations to generate a consistent object-level geometric representation. By combining compliant, adaptive grasping with large-area tactile imaging and accurate proprioceptive mapping, the system achieves precise surface contour perception without the need for time-consuming full-surface scanning. Such capabilities are particularly advantageous in unstructured scenarios where visual sensing is incomplete or occluded, offering both high perceptual efficiency and broad application potential.

FlexiRay benchmarking via extended tea-preparation teleoperation
Currently, efficient learning of complex skills from human demonstrations has emerged as a key research direction for general robotic manipulation48–50. However, existing studies predominantly focus on capturing only visual demonstration data, which limits dexterous manipulation in contact-rich environments and robots’ ability to interact with objects exhibiting different properties. In this work, we aim to demonstrate the potential of FlexiRay in providing richer multimodal information to enhance the quality of human demonstration data, using an extended-duration teleoperation task of traditional tea preparation as a case study.

This task integrates fine manipulation, in-hand tool reorientation, thermal discrimination, and stable pouring into a single continuous operation, which is not merely a procedural demonstration but a systematic assessment that raises high requirements on the gripper’s multimodal sensing capabilities. Within an uninterrupted sequence, the system unifies force sensing, contact localization, proprioceptive deformation mapping, and temperature detection to underpin task execution. The tea-preparation task further acts as a controlled yet challenging platform for validating such multimodal perception. It demands the operator adapt to unpredictable object dynamics (e.g., loose tea leaf movement, liquids loshing), discriminate visually similar objects via temperature, and execute precise pouring, all under remote teleoperation, where direct physical contact is absent and visual occlusion occurs frequently. In this scenario, FlexiRay redefines conventional vision- and force-dominated teleoperation, transforming it into a multisensory manipulation process. By leveraging its inherent compliance and safe physical interaction, the gripper enables operators to make real-time decisions at every stage, supported by continuous feedback from its integrated sensing modalities. Specifically, it requires the operator to adapt to unpredictable object dynamics (e.g., loose tea leaf movement, liquids loshing), discriminate visually similar objects via temperature, and perform precise pouring, all under remote teleoperation, where direct physical contact is absent and visual occlusion occurs frequently.

In the reorientation stage, FlexiRay leverages precise contact localization and force feedback, allowing the operators to adjust the teapot tray’s pose without releasing it. For instance, the initial grasp positioned the tray plane at an ~80° offset relative to the gripper’s z-axis, which is suboptimal for pouring. Conventional teleoperation, due to visual occlusion or the lack of accurate estimation of in-hand object pose and contact distribution, necessitates large, unstable arm movements or complete regrasping. In contrast, FlexiRay enables controlled micro-sliding against external supports, with visibility and continuous monitoring throughout. Real-time tracking of contact texture orientation and contact point trajectories prevents slippage, facilitating a smooth 25° in-hand reorientation (Fig. 9c, h) and allowing pouring from a mechanically advantageous arm configuration. In the thermal identification step, FlexiRay’s temperature-sensitive sensing layer delivers real-time feedback via colorimetric change of markers (Fig. 9e), supported by a temperature-sensing model that estimates the contact-region temperature. This integrated approach allows operators to reliably discriminate between room-temperature and hot water cups during remote teleoperation, eliminating the need for repeated verification. As shown in Fig. 9j, both the chromatic variation and the predicted temperature profile clearly separate cold and hot contact intervals, providing quantitative temperature estimates. These results demonstrate FlexiRay’s functional temperature-sensing capability in realistic applications, significantly augmenting the performance of teleoperation systems with a sensing modality seldom available in conventional setups. Notably, FlexiRay’s proprioceptive sensing stably predicts and visualizes the deformation of its compliant beams throughout operations, delivering real-time data on interaction dynamics and grasp quality, and remains fully operational even under visual occlusion. Figure 9g–k depicts force, contact position, and proprioceptive signals (encoded as curvature of front and back beam nodes) across the manipulation sequence. Larger curvature values correspond to stronger grip forces, consistent with the passive deformation properties of the Fin Ray structure. Figure 9l illustrates predicted gripper deformation at three key moments (0 s, 50 s, 170 s), offering intuitive guidance for refining grasp strategies.

Long-horizon teleoperation experiments clearly demonstrate FlexiRay’s multimodal perception capabilities, which integrate all five sensory modalities to assist human operators in achieving more precise and safer manipulation. Even under conditions of visual occlusion or partial visual information loss, FlexiRay enables reliable task execution, highlighting its potential to enhance robotic systems in adapting to dynamic and complex real-world environments. These experiments substantiate the practical advantages of multimodal sensing, showing how the fusion of force, temperature, texture, contact localization, and proprioception leads to highly adaptive and efficient robotic systems. Looking forward, integrating FlexiRay with imitation learning frameworks is expected to enable safer and higher-quality transfer of human demonstrations to robotic systems, paving the way toward more generalizable robotic manipulation.

Methods
Optical system implementation
• Camera. A camera with high-resolution image acquisition capability (12 million pixels) is selected, featuring a compact form factor (8×8.5×5 mm) and a wide-angle field of view (135°). Its compact form enables integration into the gripper, capturing high-clarity internal imagery with sufficient detail of contact interfaces.
• Illumination. Flexible RGBW (red, green, blue, white) LED lights are embedded within the silicone gel in a series-connected configuration during the fabrication process. To address the issue of ambient light interference, the system incorporates interlaced crossbeams. This beam design not only effectively mitigates interference from external light sources but also retains the flexibility required for compliant and adaptive interactions.
• Mirrors. The mirrors are strategically positioned and mounted at a specific angular orientation on the finger’s skeletal structure to extend the visual field of view. This configuration ensures comprehensive visual coverage of the interaction area even when the finger undergoes significant deformation during operation. To preserve the compliance of the finger structure without introducing structural constraints, the rigid planar mirror adopts a T-shaped design. This geometric configuration minimizes the bonding area with the flexible back beam of the finger, thereby reducing mechanical interference with the beam’s deformation. Simultaneously, the T-shaped structure ensures that the effective reflective surface area remains sufficiently large to maintain the required optical path efficiency.

Skeleton and tactile sensing pad implementation
Finger. The Fin Ray finger, fabricated from TPU materials, exhibits intrinsic compliance that enables it to passively conform to the contours of various objects without relying on active actuation mechanisms.

Tactile Sensing Pad. The tactile sensing pad primarily consists of five functional layers. The light-shielding layer is fabricated by blending Smooth-On Inc. 00-30 silicone with black color paste, aiming to block the interface from environmental light. The temperature-sensing layer is formulated by incorporating three thermochromic pigments into the silicone, leveraging the pigments’ reversible color-changing properties in response to temperature variations to enable non-intrusive thermal detection. For the reflective layer, silicone is mixed with aluminum powder and aluminum flake in a mass ratio of 400:20:3. The elastic layer utilizes a low-hardness transparent silicone (Shore 0A), whose ultra-soft mechanical properties facilitate precise transduction of surface texture information through deformations. Additionally, PDMS is employed as an elastic support layer, designed to maintain the overall compliance of the sensing pad while providing structural reinforcement to preserve deformation capacity during interactions with objects.

Fabrication and manufacturing
Our primary objective is to endow FRE with rich haptic perception capabilities through a vision-based tactile sensing method while preserving its inherent compliance and passive deformation characteristics. To achieve this, the conventional acrylic support structure is replaced with PDMS material, and flexible LED strips are adopted for illumination. The fabrication process is detailed as follows:

First, a reflective layer is fabricated by mixing Smooth-On Inc. 00-30 silicone with aluminum powder and flake according to the specified mass ratio. To ensure surface quality, the mixture is poured onto a glass substrate, spread to form a thin film (thickness: 0.25 mm) with a squeegee applicator, and cured at 80 °C for 30 minutes. Subsequently, an array of round markers is removed from the cured film using a laser cutter. Next, a temperature-sensing layer is prepared by mixing the silicone with three thermochromic pigments in a mass ratio of 20:1:1:1. These pigments exhibit distinct thermochromic responses: one transitions from colorless to blue at 18 °C, another from green to yellow at 38 °C, and the third from colorless to magenta at 60 °C. Consequently, this combination enables a temperature-dependent color transition of the thermometric layer: it appears pink above 60 °C, yellow between 38 °C and 60 °C, deep blue below 18 °C, and dark green under ambient conditions. Using the same application method, a 0.15 mm thick temperature-sensing layer is produced and bonded to the reflective layer. The temperature-dependent color of the thermosensitive silicone is shown in Supplementary Fig. 7. To avoid the interference from ambient light, a 0.15-millimeter-thick black light-shielding layer is applied by mixing silicone with black pigment paste and coating onto the previous layer.

The prepared composite layer is then removed and placed into a mold. A low-hardness, transparent silicone (mixed in a ratio of 1.8:1) is poured over it to create a soft, elastic layer with a Shore A hardness of 0, approximating that of human skin. To ensure a uniformly smooth surface, the material is smoothed out with a squeegee and cured at room temperature for 90 minutes. Subsequently, the RGBW LEDs are connected in series and affixed to the TPU front beam. PDMS is mixed in a ratio of 5:1 to form a transparent support structure capable of withstanding large deformations. The TPU front beam with integrated LEDs is positioned within the mold. Further, PDMS is poured to encapsulate these components, resulting in a tactile sensing pad capable of perceiving texture, temperature, and other contact information.

Following the fabrication of the tactile sensing silicone pad, the assembly of the FlexiRay is completed by integrating the mirrors, internal camera, side beams, finger framework, and the soft sensing pad. To validate its perception and grasping capabilities, a gripper is designed comprising two identical FRE fingers. A single stepper motor actuates the gripper through linkage mechanisms, enabling a substantial range of opening and closing motions.

CMA-ES based optics layout optimization
The equilibrium between finger compliance and the robustness of the tactile sensing region presents a significant challenge in the advancement of soft VTS. To mitigate the limitations of FOV and occlusion blind spots associated with large-deformation finger structures, a multi-mirror FOV optimization method is proposed, leveraging CMA-ES52,53. Through the optimization of both structural and layout parameters of the camera and mirrors, a visual reflection system composed of multiple planar mirrors is established on the backside of the finger. This configuration facilitates a single camera in achieving comprehensive coverage of the entire deformation range and contact region of the Fin Ray, thereby enabling enhanced vision-based haptic perception.

Initially, the displacement of nodes under the applied force F is collected for the Fin Ray, which has identical structural parameters. The displacement is represented by the coordinates of the lateral 2D cross-section, with the back joint nodes denoted as {Ni}ni=1 and the tactile sensing area joint nodes labeled as {Pi}ni=1. Consequently, a mapping from force to deformation is established, represented as f :F→{Ni,Pi}ni=1. The subsequent part provides a detailed account of the construction and solution of the optical layout optimization problem.

Decision variables
x= {θm i ir, tm i ir, lm i ir}ni=1 ∪{u, ϕ} ⊂ R (1)
where {θm i ir, tm i ir, lm i ir} represents the parameters associated with the mirrors. Specifically, θm i ir denotes the angle between the mirror and the side, tm i ir indicates the midpoint offset distance, and lm i ir signifies the length of the mirror. The camera is positioned along the baseline defined by the backside and the base nodes of the tactile sensing pad. The distance from the camera to the back base node is expressed as a coefficient u, relative to the length of the baseline. Furthermore, the angle between the optical axis of the camera and the baseline is represented by ϕ.

Objective function
With a fixed FOV, the discrete rays produced by the camera can be represented as R={rj}mj=1. The primary objective of the optimization process is to maximize the coverage of the tactile sensing region by the collection of FOV rays across various deformations. This goal can be achieved through two approaches: direct imaging and single reflections from mirrors.

By sampling K distinct loads from the force-displacement mapping f, a set of joint nodes D is obtained. Each deformation structure corresponds to a discrete set of target points p, denoted as Pd, within the tactile sensing region. The radius of the illumination range for a single FOV ray is defined as R/lc, which depends on the propagation distance lc. Consequently, the objective function can be expressed as follows:
Maximize f(x)= 1/K ∑d ∑r ∑p I(x, r, p, R) (2)
where I(x, r, p, R) is an indicator function that returns 1 if the target point resides within the coverage range of the ray beam; otherwise, it returns 0.

Constraints
The following constraints must be adhered to:
• Geometric constraints. The lengths, offsets, and rotation angles of each mirror, as well as the position of the camera, must remain within predefined ranges: lm i ir ∈ [l min, l max], tm i ir ∈ [t min, t max], θm i ir ∈ [θ min, θ max], u ∈ (0, 1).
• Occlusion constraints. The potential occlusion between mirrors during the deformation process must be taken into account. If an occlusion exists along the line of sight, the indicator function I(x, r, p, R) returns 0.
• Safety constraints. Under all deformation conditions, mirrors must not interfere with the front and back beams. If this condition is not met, the indicator function I(x, r, p, R) returns 0 as a penalty.

Solution process
The optimization process utilizes the CMA-ES algorithm to search for optimal layout solutions by sampling candidate solutions from a multivariate Gaussian distribution. The multivariate Gaussian distribution is defined as N(μ, σ2C), where μ is the mean vector, C is the covariance matrix, and σ represents the step size.

In the y-th generation of the optimization process, for a population size of β, the candidate solutions {xi}βi=1 are obtained by sampling from the distribution N(μ(y), σ(y)2C(y)):
xi=μ(y)+σ(y) C(y)si (3)
where si ∼ N(0, I), with I denoting the identity matrix. The solutions are updated with the following equations:
p(σ y+1)= (1−c σ)p(σ y)+ √c σ(2−c σ)λ w C(y)−1 (4)
p(c y+1)= (1−c c)p(c y)+ √c c(2−c c)λ w H(σ y+1) (5)
where c σ and c c are cumulation factors, respectively. H(σ y+1) is the Heaviside function, and δ= ∑η i=1 wi C(y)si, λ w=P η i= 1 1 w2 i, ∑η i=1 wi=1.

Therefore, the parameters of the multivariate Gaussian distribution for each generation are updated as follows:
μ(y+1)=μ(y)+c l σ(y) C(y)si (6)
σ(y+1) =σ(y)exp(c σ/dσ (||N(0, I)||/E||N(0, I)|| − 1)) (7)
C(y+1)= (1+c 1 c μ − c 1 − c μH(σ y+1)) C(y)+c 1 Δ 1 + c μΔ μ (8)
where Δ 1 and Δ μ correspondingly denote the rank one and μ updates. The parameters c l, c 1, and c μ represent the learning rates for the mean, rank one, and μ updates, respectively. The damping factor d σ is used for the adaptive accumulation of step sizes.

Starting with initial parameters that satisfy the constraints, the parameters of the multivariate Gaussian distribution are updated iteratively. This process continues until either the expected stopping condition for the objective function is met or the maximum number of optimization generations is reached. The optimized camera and mirror layout parameters are then derived from the distribution. Sensitivity analysis of the decision variables and hyperparameter analysis of the optimization algorithm are detailed in the Supplementary Fig. 1 and Supplementary Fig. 2).

Proprioception data collection platform and procedures
The dataset collection platform for proprioception and thermoreception tasks is shown in Fig. 10a. FlexiRay is mounted at a fixed position on an optical bench, with the reference coordinate system established at the lower-left corner of the tactile sensing pad (Fig. 10c). A six-axis force sensor is attached to the end of a UR5e robotic arm, whose tool coordinate system is aligned such that its z-axis is parallel to the z-axis of the reference frame, enabling normal force application. A 3D-printed contact probe is mounted on the sensor, designed to simulate diverse contact conditions through 12 interchangeable geometries, including hemispherical, square, triangular, and cylindrical tips (Fig. 10e, full set in Supplementary Fig. 5). During data collection, the probe is randomly positioned within the feasible contact domain in the x–y plane, randomly rotated around the z-axis, and then advanced along the z-axis until contact is detected by the force sensor, which triggers synchronized acquisition. At this stage, three data streams are recorded simultaneously: lateral beam deformations captured by dual global cameras (Fig. 10b), haptic images from the internal camera, and force readings from the load cell. In parallel, proprioceptive node trajectories are extracted in real time (Fig. 10d), while the robotic end-effector position is continuously logged to ensure precise temporal and spatial alignment across all modalities. For thermoreception tasks, two specialized probes are employed: a heating probe and a cooling probe (Fig. 10f). The heating probe incorporates a thermocouple at the contact surface for ground-truth temperature measurement, backed by a thin heating film. The cooling probe similarly includes a thermocouple at the interface, followed by a semiconductor cooling sheet, a cooling fin, and an active air-cooling path for thermal dissipation. During thermal sensing trials, the contact temperature measured by the thermocouple is recorded synchronously as reference data.

Machine learning implementation
In term of model implementation, the PP-LiteSeg semantic segmentation model is built upon the STDC2 backbone, with the objective function using Cross-Entropy Loss. The model is optimized using stochastic gradient descent with momentum set to 0.9 and weight decay of 4×10−5. A polynomial decay learning rate scheduler is employed, with an initial learning rate of 0.01 and a decay factor of 0.9. After training, the model’s performance evaluation shows a mean Intersection over Union of 89.98%. The proprioceptive sensing model consists of four output heads that respectively predict normal contact force, contact position, right-side node localization, and left-side node localization. Each head includes two fully connected layers with 512 and 256 neurons, followed by output layers of dimensions R1, R3, R28 and R28, respectively. All outputs are trained with the mean squared error (MSE) loss, and the learning rate is set to 1×10−3. The dataset contains 31,904 haptic images samples, divided into training and validations sets with a 4:1 ratio. For temperature prediction, the marker extraction model is implemented based on the YOLO11 framework and a U-Net architecture with a ResNet50 encoder is employed. The model takes RGB three-channel images as input and outputs a single-channel grayscale temperature field map. The loss function is MSE, with a learning rate of 1×10−4. The dataset for temperature sensing includes 17,309 temperature-aligned haptic images samples, also split into training and validations sets at a 4:1 ratio. Additionally, the classification head of the texture recognition model is composed of fully connected layers with 512 and 256 neurons. The loss function used is Cross-Entropy Loss, and the model is trained with the Adam optimizer, with a learning rate set to 1×10−3. Additionally, all experiments are conducted on a workstation equipped with an Intel Core i7−12700KF CPU and an NVIDIA GeForce RTX 4070 GPU (12 GB VRAM). Upgrading computational resources can further improve training efficiency and reduce total training time.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This paper proposes a soft gripper with vision-based tactile sensing and utilizes the multiple mirrors to expand their\nperception area. The authors present related experiments on force estimation, gripper shape reconstruction, contact points\nestimation, texture perception and temperature sensing. This paper demonstrates the potential of the combination between\nsoft robots and vision-based tactile sensor. However, the reviewer has some comments on the contribution and technical\nmethods in the current presentations.\nMajors:\n1.The proposed structure is fin-ray based structure, which is widely used in the soft gripper. The design novelty of this paper\nshould be made clearer, such as the multiple mirrors integrations (for large perception rage) or the temperature sensing\nability (for multi-modality)? The novelty is not clear. Besides, the authors claimed that previous design is rigid, but this paper\ndose not present the advantage of the proposed compliant design?\n2．The proposed sensor provides human-like multimodal deep learning framework to decouple contact forces, position,\ntexture, temperature, and proprioception, but some of these abilities have not been fully explored. For example, the\napplication of contact position estimation. Besides, the texture is the intrinsic ability of major camera-based tactile sensor,\nwhich may not be included into contribution?\n3. For the bio-inspired design of the FlexiRay, the authors do not explain the relationship between the biostructure and the\nproposed design, such as the mirrors independent design (also I am confused with figure 1). Besides, there is large space\nfor the CMA-ES based optics layout optimization, but the reviewers can’t find the effeteness of each parameters and the\ncomparison of optimization.\n4. The authors claim that the FlexiRay can reproduce 5 kinds of sensory modalities of the human hand and demonstrate the\ncapabilities independently through a series of experiments. The authors developed a series of learning models to decouple\ndifferent sensory tasks, but how the multi-modality sensory information can be used together to achieve more difficult tasks is\nnot demonstrated in the article. A more complicated task with more sensor information fusion is suggested to show the\nadvantages of the five modalities of the FlexiRay.\n5. The presentation of each image is not clear, such as the Fig 5 and Fig 6 are both involved with texture perception.\nBesides, in supplementary video 4, the FlexiRay is used to classify different kinds of balls and place them in corresponding\npositions. However, from the video, it seems that the FlexiRay simply grasps the object, stays for a while, and puts it on the\nground. The classification process isn’t shown clearly, so this video needs to be polished to be more persuasive. The Fig.4\nis filled with force, position and reconstructions. The authors demonstrated that by touching a cylinder with a slight angle of\ninclination, the FlexiRay can reconstruct the 3d morphology of the given object. However, the object chosen in this\nexperiment is too simple. Can this be used to reconstruct objects with more complex shapes, for example, shapes with\nsharper edges, non-convex objects, or objects with obvious change in curvature (e.g., gourds)?\nMinors:\n1.Typos: in line 37, often-often; in line 134, functionaln-functional. The paper should be carefully revised to avoid these\nquestions.\n2.Tab 1 is not comprehensive.\n3.Temperature perception lacks quantitative results.\n4.Image process pipeline need to be explain.\n5.What is the “low-hardness, transparent silicone” in L442? Please provide details of all materials. How to ensure the\nreflective layer surface quality in fabrication?\n6.The estimation error is relatively large, please explain the reason. (0.81mm contact location error, 0.135 N force errors). It\nis interesting that the proprioception accuracy is higher than contact positions. Science the variables of location is smaller\nthan proprioception, but they use same feature in Fig 3 b.\n7.In Fig. 4a, the authors use a grasp experiment in which the FlexiRay is used to grasp different objects by a constant force\nto show the force estimation capability of the FlexiRay. However, I don’t think this can be called dynamic force estimation.", "reviewer_2": "The work titled “Tactile-integrated FlexiRay: Breaking Planar Limits by Harnessing Large Deformations for Flexible, Full-\nCoverage, Human like Multimodal Sensing” is indeed an interesting work. This work addressed the most critical challenges\nin modern day robotics. The sensing in robotics is vital due to its complexities. Even though I appreciated the work and the\nresults, I would like to point out a couple of comments to improve the clarity of the presentation.\n1. The authors have presented the proposed FlexiRay can be incorporated five of the seven sensory modalities found in the\nhuman hand. This is really cutting edge if you can demonstrated more details for the claim. For example, what is the latency\nof each of them? Can you provide a table and integrate to the paper?\n2. The table 1 shows the comparison. Did you come across HaptiTemp? If not, I would highly recommend to take a look.\n3. It has been mentioned that FlexiRay’s classification model accuracy is 95.83% compared to GelSight mini (94.17%).\nGelSight has been evolved among the different research groups in different part of the world. Would you consider this claim\nis valid for up to date state of art of GelSight? Moreover, the difference that you presented is very close. It could be the\nsamples. I would suggest you to rethink about the claim and articulate accordingly.\n4. Is the temperature changes linear? Hot to cool and cool to hot? Please add them as results.\n5. Multimodal perception is something very exciting in the current trajectory of the robotics and AI. However, the integration\nand communication are really challenging in tele-operation. What sort of communication you would suggest? Please add to\nthe paper.\nIn general, this is quite interesting work. I would like to see how the authors would shape the paper for publications."}, "author_response": "Dear reviewers,\nWe thank the reviewers for all the constructive comments. This manuscript is the revision\nof our original submission entitled “Tactile-integrated FlexiRay: Breaking Planar Limits\nby Harnessing Large Deformations for Flexible, Full-Coverage, Human-like Multimodal\nSensing” (Manuscript ID: NCOMMS-25-06676A).\nIn response to the comments, we have significantly revised our manuscript, conducted\nadditional experiments following the reviewers’ suggestions, and provided detailed\nresponses point-by-point in this document. In the rest of this document, we first summarize\nthe key revisions in the revised manuscript, and then we reply to each comment\nindividually with more details. For better readability, we highlighted in blue the changes in\nthe revised version of the manuscript. The key revisions are as follows.\n• Hardware upgrade: FlexiRay adopts a redesigned RGBW LED layout to enable\nclearer texture visualization, alongside a new thermochromic pigment mixture\nformulation for wider, more distinguishable color transitions across temperature\nvariations. A quantitative characterization (15–70 °C) is added in the Supplementary\nFig. S7. Therefore, all experiments are fully repeated with the upgraded hardware.\n• Enriched dataset with enhanced generalizability: For the upgraded hardware, we\nfurther expanded the dataset’s diversity and scale. The new dataset includes 12 probes\ninstead of 4, with interactions exceeding 40,000 and validation samples increased to\n5,000, which further enhances model robustness and generalization. Additionally, the\nmodel now outputs synchronized dual-side nodes, enabling more accurate\nproprioceptive perception for spatial contact reconstruction.\n• Further illustration on design optimization and multi-modal perception: More\nsystematical analysis of the structural optimization algorithm is added in the\nSupplementary Fig. S1 and S2. Besides, the image-processing pipeline is clarified,\nwith hard-synchronized outputs and ~25.8 ms latency (30 Hz) ensuring real-time\nresponse.\n• New interaction frameworks and experiments: We propose an admittance-control\nhuman-robot interaction (HRI) framework for real-time gripper-arm interaction,\nvalidated via feather-contact tests etc. to further confirm FlexiRay’s subtle tactile\nperception and enhanced safety. Additionally, handwriting trajectory prediction has\nbeen extended to 4 paths, and local shape reconstruction now encompasses more\ncomplex objects. Finally, we present FlexiRay as a more intelligent universal\nmanipulation interface, with validation through an extended tea-preparation\nteleoperation experiment. Here, FlexiRay delivers richer sensory feedback to the\nremote operator (force, contact, proprioception, temperature), supports long-horizon\ntasks, and improves stability and efficiency under occlusion or degraded vision.\nNotably, by leveraging richer multi-modal tactile perception, FlexiRay exhibits\nsignificant potential in further combination with current development in imitation\nlearning and to further enable higher-quality human demonstrations, with the ultimate\ngoal to improve the dexterity and safety of robotic systems.\n• Writing and more comprehensive comparison: Contributions have been\nreorganized for enhanced clarity, with related studies and the comparison table\nexpanded to more objectively and comprehensively highlight the improvements of the\ncurrent work; the overall presentation has also been refined. Additionally, the text has\nundergone thorough checking and proofreading, with language expression further\noptimized.\nWe believe that our work would be of substantial novelty and appealing in terms of soft\nrobotics design and perception, and Human-Robot Interaction. We declare no conflict of\ninterest and that the paper is not under consideration elsewhere. We appreciate your\nconsideration of our manuscript, and look forward to receiving comments from the\nreviewers. Please do not hesitate to let us know if you have any questions or suggestions.\nYours sincerely,\nAuthors"}, {"round": 2, "reviewer_comments": {"reviewer_1": "General Comment:\nIn this revision, the authors have highlighted the contribution of this work and their sensing capabilities in complex tasks,\nwhich has partly addressed my concerns. However, some issues require further improvement before considering\npublication.\nSpecific Comments:\nComment 1:\nThe authors studied the temperature-related color changes of the thermochromic layer in the revised manuscript.\nThe resolution, response time, and repeatability are basic performances for temperature sensors that still lack characterization.\nHow to obtain the temperature mapping from vision-based images? The temperature sensing in Fig. 9e seems just show a\ntemperature-sensitive film without further analysis. Will the temperature sensing couple with and influence force sensing or\nother modalities?\nComment 2:\nThe illustration in Fig. 1a is still confusing and may cause some misleading. Although this work reproduces part of human\nsensing capability, the structure design is far from the biological skin. It is far-fetched to directly correspond distributed\ncutaneous receptors or tendons with the designed structures. Additionally, the clarification of “seven perceptual modalities”\nseems unusual, especially integrating proprioception into tactile perception.", "reviewer_2": "The authors have successfully addressed my comments with strong scientific evidence, which has been incorporated into\nthe revised manuscript.\nOverall, I am satisfied with the modifications and amendments they have made."}, "author_response": "Dear reviewers,\nWe thank the two re viewers for all the constru ctive comments. Thi s manuscript is the\nrevision of our original submission entitled \"Tactile-integrated FlexiRay: Breaking\nPlanar Limits by Harnessing Large Deformations for Flexible, Full-Coverage, Human-like\nMultimodal Sensing\" (Manuscript ID: NCOM MS-25-06676A).\nIn response to the comments, we have significantly revised our manuscript, conducted\nadditional experiments following the reviewers’ suggestions, and provided detailed\nresponses point-by-point in this document. In the rest of this document, we first summarize\nthe key revisions in the revised manuscript and then we reply to each comment individually\nwith more details. For better readability, we highlighted in blue the changes in the revised\nversion of the manuscript. The key revisions are as follows.\n• Establishment of a new temperature perception model: We enhanced the\ntemperature perception model from a categorical (cold/warm/neutral) classifier to a\npixel-wise continuous temperature field regression model, capable of providing both\nmarker-level and contact-region-level temperature estimates.\n• Comprehensive temperature perception quantitative evaluation: We added\nsystematic experiments to evaluate the model’s accuracy, resolution, response time,\nand repeatability. The model achieves an average absolute error of 1.17 °C and a 95th\npercentile error of 3.34 °C in reconstructing the 3D distributed flexible tactile markers’\ntemperature field.\n• Cross-modal coupling analysis: We further analyzed and quantified the influence of\nthermal variation on other sensing modalities. Results show that color transition in the\nthermochromic layer exerts minimal coupling effects on force, contact localization,\nand proprioceptive perception, validating the robustness of the multimodal sensing\nframework.\n• Optimization of figures and textual clarity: We revised Fig. 1 to eliminate ambiguity\nand improve conceptual clarity. Fig. 9 has been optimized to include numerical\ntemperature prediction results in the extended tea-preparation teleoperation task.\nSimilar revisions have been made throughout the manuscript to improve readability\nand precision.\n• Clarification of Modalities: Ambiguities regarding sensing modalities were\nidentified in the original manuscript, particularly the conflation between tactile sensing\nand kinesthetic sensing (proprioception). Following a comprehensive literature review,\nwe have redefined the five proposed modalities under the term \"haptic perception\ncapabilities.\" Specifically, we retain the conventional terminology \"tactile sensing\"\nwhen referring exclusively to the detection of pressure, contact location, texture, and\ntemperature. In contrast, \"haptic perception\" is employed to describe scenarios\ninvolving both proprioception and tactile sensing. Accordingly, we have revised the\nmanuscript title to \"FlexiRay: Breaking Planar Limits by Harnessing Large\nDeformations for Flexible, Full-Coverage, Human-like Multimodal Haptic\nPerception\" and standardized the consistent use of these terms throughout the text.\nWe believe that our work would be of substantial novelty and appealing in terms of soft\nrobotics design, perception, and Human-Robot Interaction. We declare no conflict of\ninterest and that the paper is not under consideration elsewhere. We appreciate your\nconsideration of our manuscript, and look forward to receiving comments from the\nreviewers. Please do not hesitate to let us know if you have any questions or suggestions.\nYours sincerely,\nAuthors"}]

Decision letter