Sensor fusion of touch & vision in soft manipulators for fruit picking

Domain: biology
Published: 2026-03-23
Editorial outcome: major_revision

Abstract
Modern agricultural robotic systems are constrained by limited sensing and manipulation capabilities, particularly in fruit harvesting, where variability in color, size, and firmness poses significant challenges. Existing solutions, often reliant on rigid grippers and single-modality sensors, frequently cause fruit bruising and substantial post-harvest losses. Here, we present a compact, five-finger soft robotic gripper with integrated multimodal sensing—including vision, tactile, and curvature sensing—for adaptive and non-destructive fruit harvesting. The system incorporates 13 sensors, onboard electronics, local computation, and a rotational harvesting module. Each finger embeds custom stretchable optical fibers functioning as tactile and curvature sensors, while the palm houses a miniaturized camera and distance sensor. The gripper actuates within two seconds at 80 kPa, exerts up to 6 N of pulling force, and lifts objects up to 1 kg—more than 16 times its own weight. Its workspace expands from 200 mm² to 14,000 mm², enabling the handling of fruits with diverse shapes and sizes. Each finger bends up to 240°, with performance closely matching finite element predictions. For vision measurements, the hue channel in the HSV color space enables robust real-time color detection, achieving 100% shape classification accuracy and a size measurement error below 1.8%. Tactile sensors distinguish soft from firm objects, while curvature sensors accurately measure the finger’s bending state—both based on optical signal loss. Real-time demonstrations validate the system’s ability to assess ripeness using multimodal data (vision, tactile, and curvature) and successfully harvest greenhouse strawberries with minimal damage. This platform offers a versatile, sensor-rich solution for both precision agriculture and general-purpose robotic manipulation.

Introduction
Many fruits ripen at varying rates and are primarily inspected visually, relying on changes in skin color to determine optimal picking times for farm-to-market distribution. Some crops have acceptable quality for commercial harvest over a period of weeks (e.g., citrus), whereas others must be harvested within a day or two of achieving acceptable quality (e.g., strawberries, raspberries). Fruit harvested too early will not have good flavor and sweetness, whereas fruit harvested after the peak of ripeness may be too soft and spoil quickly. 1-4. Fruits that develop over a long period (e.g., pomegranate, citrus) are typically firmer, often have thicker skin, and are easier to inspect visually due to their slower spoilage rates. In contrast, fruits with shorter ripening periods (e.g., berries) are usually soft, often have thinner skin, and are more challenging to inspect visually because of their rapid transition from acceptable quality to spoilage. These differences contribute to substantial variation in postharvest losses in the supply chain, with approximately 6–7% loss for pomegranates and up to 25% for strawberries 4-7.

Visual assessment of fruit quality is not always feasible, as many fruits exhibit minimal or inconsistent changes in skin color that may not be associated with ripeness. For example, avocados and pawpaws do not reliably change from a green skin color during ripening and vary in ripening time. Avocados can be of an acceptable quality for harvest over a period of weeks, whereas pawpaws become too soft in days after they ripen, leading to high spoilage rates during distribution. Postharvest losses of avocados can reach up to 24% 8, and pawpaws, despite being delicious, are only distributed locally because of their short storage life and ease of bruising.

Tactile inspections are an alternative to visual assessments of ripeness using variables such as firmness and texture. Visual inspection is faster and primarily based on color observation, 9,10 but it is highly dependent on lighting conditions. Conversely, tactile inspection offers precise measurements of mechanical properties but is slower and requires careful fruit manipulation, making it less suitable for fruits with rapid spoilage rates, soft skin, and a tendency to bruise. Insidiously, many fruits soften when ripe and become subject to bruising, making tactile inspection a delicate process that contributes to spoilage.

Fruit inspections are critical for harvesting decisions. Traditionally, these decisions have been made by human workers, leading to challenges such as labor shortages, worker health concerns, and inaccuracies in picking, which exacerbate spoilage during pre- and post-harvesting stages 11-13. In recent years, harvesting robots with integrated inspection capabilities have emerged as a promising technology to reduce spoilage and improve distribution efficiency 14,15. However, most current robotic systems are designed for indoor applications, even though over 99.9% of crops are cultivated in outdoor environments, with greenhouse farming comprising less than 0.1% of global cropland area 16,17. These robots typically use robotic arms and grippers, often integrated into wheeled platforms, with inspection systems based on either visual or tactile sensors.

Vision sensors such as RGBD, CMOS, thermal, and multi-hyper-spectral cameras are employed to gather data and assess fruit ripeness based on color 9,15,18-20. Similarly, tactile sensors, integrated into robotic grippers, often measure changes in deformation through technologies such as conductive elastomers, fabrics, piezoelectric sensors, optical fibers , cameras, metal wires, liquid metals, and carbon nanotubes 21-33. These sensors detect changes in capacitance, resistance, light loss, deformation, or magnetic flux to infer fruit characteristics 31,34- 37.

Current harvesting robots face several challenges when using these sensors. Robotic visual inspections primarily rely on external or head-mounted cameras integrated into robotic arms or platforms38,39. These systems impose significant limitations on visual servoing capabilities due to occlusion, partially hidden objects, and low accuracy in detecting color, size, and shape 40-42. Lighting conditions can also cause detection errors of up to 50% 41. Robotic tactile inspections often use advanced materials such as conductive elastomers, optical fibers, or carbon nanotubes integrated into grippers. These systems, however, face scalability issues, uneven sensor distribution across gripper surfaces, and integration complexities 34, reducing the reliability of measurements and hindering direct physical interaction with fruits 43. Moreover, many robotic systems lack the ability for simultaneous close inspection, as their sensors are often bulky and integrated into rigid grippers, increasing the risk of bruising during pre- and post-harvesting stages 26,38-40.

One method of addressing these challenges fuses sensing capabilities from simultaneous high-resolution visual inspection and distributed tactile feedback. Combining tactile and visual measurements provides a more accurate assessment of ripeness and other characteristics such as shape and size. Soft robotic systems made of elastomeric materials, which inherently enable gentle interaction, permit tactile assessment and picking that addresses bruising issues, improving harvesting decisions, and reducing spoilage caused by inaccurate or time-consuming measurements 44-48.

This integrated approach enhances sensor utility by leveraging each sensor’s strengths to compensate for its limitations, making the technology applicable to a broader range of fruits and environmental conditions. For instance, tactile inputs can verify visual measurements in poor lighting conditions or for unfamiliar fruits, and vice versa. Such advancements can increase the variety of fruits distributed, reduce spoilage, and lower supply chain costs by enabling faster and more accurate harvesting decisions.

To achieve high-resolution visual and tactile measurements for rapid harvesting, integrating a collocated vision system and stretchable optical fibers provides a pathway to develop compact, scalable grasping solutions without complex integration challenges 22,23. Fixed camera approaches are prone to self-occlusion, whereas moving camera collocation addresses this issue, significantly improving visual dexterity and enabling the extraction of high-resolution features. However, moving-camera collocation has primarily been applied in pose estimation and collision avoidance, with limited exploration in fruit ripeness estimation 34,42.

Existing grippers with integrated tactile or vision systems typically focus on a single sensing modality or offer limited multimodal integration49. For instance, most vision systems are primarily designed for object detection but do not provide additional outputs such as shape or size information that is especially useful for tasks like fruit picking50-52. Similarly, grippers are designed only to rely on tactile sensors measurements, which often tend to be bulky relative to the gripper’s size and limited manipulation dexterity34-37,46,47,53-58. This imposition causes design constraints, requiring larger grippers that compromise flexibility and workspace. Beyond vision and tactile, only a few grippers incorporate curvature (proprioceptive) sensors with tactile feedback to estimate the bending state and touch59-66. Even when present, these sensors are typically not integrated with other sensing modules, despite their importance in handling dynamic objects. Moreover, most of these grippers are rigid; only a limited number of soft grippers are capable of achieving embodied multimodal sensing37. Even those are often restricted to tactile feedback, with cameras mounted externally on the main robotic platform rather than directly integrated into the gripper67-69. Enabling true multimodal sensing within a gripper is essential for achieving versatile manipulation capabilities, particularly under dynamic conditions. However, integrating a large and diverse set of sensors into a compact form factor—without compromising the gripper’s functional performance—remains a significant challenge. This becomes even more critical in agricultural tasks, where object characteristics such as shape, size, color, ripeness, and weight can vary substantially. A soft gripper with embedded tactile, vision, and curvature sensors that collectively provide diverse sensing modalities would significantly enhance manipulation performance in such variable environments as exist in biological systems, which have not yet been fully realized into soft grippers.

Previously, we have developed and implemented soft optical fiber-based tactile sensors that measure light intensity loss in response to deformation or force. By implementing these sensors in a manipulator, we were able to demonstrate the prediction of ripeness in fruit (i.e., tomatoes) from tactile perception without vision, but imprecisely .22

We fused low-cost, camera-based vision and fiber-optic-based tactile inspection into a five-digit (radially symmetric) soft gripper featuring underactuated control, rapid actuation, a large workspace, and high load-carrying capacity. It also includes a rotational module for picking fruit by twisting its stem, as well as hosting the control unit. The visual sensing uses a collocated camera and a time-of-flight sensor, enabling high-resolution measurements of shape, size, and color. For curvature sensing, we integrated stretchable fiber optics directly into each finger, while for tactile sensing we used a U-shaped fiber optic at each fingertip. Additionally, we integrated a pressure sensor to enable closed-loop grasping, allowing the gripper to assess both physical and non-physical object characteristics through a multimodal sensing approach.

The gripper is 3D printed, assembled, and then thermally fused into a monolithic structure. For vision sensing, we embedded a miniature Raspberry Pi camera and an IR-based time-of-flight (ToF) sensor into the palm side of the gripper. Similarly, for touch and curvature sensing, we embedded stretchable optical fibers paired with laser diodes and phototransistors into each finger.

We actuated the gripper pneumatically, using solenoid valve actuation and pressure sensor feedback. We developed a custom PCB to read tactile, curvature, ToF, and pressure sensor data while also controlling solenoid valve actuation. We connected the PCB to a Raspberry Pi, running an algorithm that fuses the sensor data to assess the object's state and perform object manipulation and grasping. The harvesting module consists of a 3D-printed planetary gear rotational system, which is integrated with a slip ring and a DC motor. Finally, we mounted the gripper, Raspberry Pi, PCB, and harvesting module onto a six-degree-of-freedom (DOF) robotic arm.

We then characterized the manipulator’s workspace, bending angle, blocking force, pulling force, and load-carrying capacity under varying pressures and time intervals. We validated the bending angle and blocking force measurements through finite element simulations. After quantifying the manipulator’s grasping characteristics, we tested the vision, tactile, and curvature sensors. We evaluated the vision sensor by testing it on objects of six different colors (black, red, blue, green, white, yellow), four different shapes (triangle, circle, quadrilateral, and fruits), and three different sizes (3, 4, 5 cm). We examined the relationship between the finger bending angle, input pressure, and sensor output for the curvature sensors. Finally, we measured the relationship between applied force and tactile sensor output using a compression test on objects of different stiffness.

The gripper is fast, closing its workspace in under two seconds at 80 kPa; strong, lifting to 1 kg (1,600% of its weight) by applying a grasping force of 6 N; and has the ability to expand its enveloping area from 200 mm2 to 14000 mm2 to accommodate large ranges of produce size. Each finger exerted up to 1.5 N and bent up to 240°, closely matching FEA simulations. For vision tests, we identified the hue channel in the HSV color space as most sensitive to changes in ripeness level and more robust to different lighting conditions. The results reveal accurate color detection from our hue-based strategy and 100% shape prediction under fixed lighting at 8 cm, with perimeter measurement errors decreasing as object size increased. The curvature sensor exhibited a sharp light intensity drop to -7 dB beyond the 77.5° critical bending angle, while the tactile sensor effectively distinguished between rigid and soft objects, including strawberries, with light intensity changes of up to -8.91 dB under an applied force of 4 N. We demonstrate the gripper’s manipulation efficacy by picking various objects and fruits in real-time.

Results
2.1. Gripper Design and Working Principle
In Fig. 1, we present a general overview of our work. The objective is to demonstrate our gripper’s design and its multifunctional sensing capabilities. The soft gripper has five fingers arranged radially at 72o intervals. All actuators are connected to a rigid chassis called the finger holder. Each finger actuator has an isosceles trapezoidal form based on the principle of flexible fluidic actuators, featuring interconnected chambers made of soft elastomers. When pressurized by a pneumatic source, the chambers expand and push against each other, causing the finger to bend. A strain-limiting layer on the palmar side controls the bending. All five actuators attach to a pentagonal-shaped finger holder. The holder contains an internal channel that connects each finger’s air source to a single barbed connector, allowing simultaneous actuation of all fingers from one pneumatic input. Furthermore, we designed the rotational module to harvest fruits by rotating the gripper after it grasps the fruit. Continuous rotation about the gripper’s roll axis twists the fruit trusses, allowing the fruit to detach from the peduncle (Supplementary Fig. 1, Supplementary Movie 1). The gripper assembly includes both the gripper itself and a rotational module (Fig. 1a). A detailed explanation of the gripper design and the rationale behind its mechanism is provided in sections 4.1 and 4.2 of the Experimental Methods (EM), while the fabrication process is described in Supplementary Fig. 2 and section 1 of the Supporting Information (SI).

In Fig. 1b, we illustrate how tactile and curvature sensors are embedded in the gripper. The red light indicates the curvature sensor, while the blue light indicates the tactile sensor (Supplementary Movie 2). We tested the gripper on two fruits—strawberries and avocados—under different lighting conditions. From our results, we were unable to visually characterize ripeness because measurements were highly influenced by lighting conditions. The tactile feedback, however, was unaffected by lighting changes and accurately measured the ripeness of both fruits (Fig. 1c-i).

2.2. Gripper Characterization
In Fig. 2a–b, we show images of the fingers printed on the build platform, along with the corresponding material properties and stress-strain relationships. The rigid material 𝐸 = 1.7 GPa 𝛾 = 0.3) exhibits a tensile strength of approximately 40 MPa, while the soft material’s (SIL 30, 𝐸 = 1 MPa, 𝛾 = 3.5) tensile strength is about 4 MPa. Fig. 2c–d presents the finger bending angle results and corresponding FEM simulations (Supplementary Movies 3-4). Each finger can bend up to 240° at 70 kPa, which matches the experimental findings. The simulation data indicate a maximum stress of 0.887 MPa and a minimum safety factor of 3.834 at 70 kPa, corresponding to a bending angle of 245.1°. We describe the experimental and simulation procedures in EM sections 4.3 and 4.4.

Fig. 2e–f illustrates the workspace analysis with both visual and simulation representations. The maximum workspace area is around 14,000 mm² at 0 kPa, and the minimum is about 200 mm² at 80 kPa (1,500% of the original area). Images of the gripper at different pressures are superimposed alongside their simulated counterparts (Supplementary Movies 5-6). We describe the experimental procedures in SI section 2 and the simulation procedures in EM section 4.4.

In Fig. 3, we detail the gripper’s force characterization. Fig. 3a–b shows the blocking force exerted by a finger against a rigid object, reaching 1.26 N at 100 kPa. The corresponding simulation at 100 kPa indicates a maximum stress of 0.793 MPa, a minimum safety factor of 4.29, and a blocking force of 1.494 N, aligning well with experimental results (SI section 3 for test procedure, Supplementary Movie 7). Fig. 3c and Supplementary Fig. 3a-c depict the gripper’s pulling force overtime for 4 cm diameter hemispherical objects at input pressures ranging from 70 to 130 kPa. The pulling force increases with input pressure, reaching a maximum of 8.10 N at 130 kPa. Object size also influences pulling force: for a 2 cm diameter hemispherical object, the gripper achieves about 1.63 N at 100 kPa, whereas a 5 cm diameter hemispherical object yields approximately 5.18 N at the same pressure. Overall, larger objects enable higher pulling forces at increased input pressures (see Fig. 3d, Supplementary Fig. 3c, Supplementary Movie 8). Finally, vertical load capacity tests show that the gripper can easily handle up to 1 kg—1,600% of its weight (see Fig. 3e and Supplementary Movie 9). We describe detailed experimental procedures for pulling force, and load capacity tests in SI sections 4 and 5

2.3. Vision sensor characterization
In Fig. 4, we present the characterization of our vision sensor and its accuracy. Fig. 4a and Supplementary Fig. 4 show the experimental setup, which includes a studio box for consistent lighting, as well as a gripper, camera, and distance sensor positioned in the same location. Detailed information about this setup and the experimental methods can be found in EM sections 4.5 and 4.6.

We first tested the camera for color measurements. In Fig. 4b, we report the hue distribution of green, red, and strawberry fruit. The results indicate that the mean (µ) and standard deviation (σ) of the hue for the red color is 172.26±25.83, green is 70.77±2.69, and strawberries are 30.01±60.56. We also measured saturation and value distributions (Supplementary Figs. 5-6) for red, green, ripe strawberries, and other colors such as blue, yellow, white, and black. The hue for black proved difficult to measure, while the hues for blue and yellow were about 101.20 ± 1.56 and 17.27±77, respectively. From this analysis, we concluded that hue offers the clearest color differentiation.

We examined strawberries in different ripeness stages: unripe, partially ripe, and fully ripe (labeled strawberry-1, -2, -3, and -4). We observed that the two fully ripe strawberries (strawberry-1 and -2) had hue channel values of 30.01 ±60.56 and 35.45 ±65.93, making them difficult to distinguish from each other due to the large variation in red hue channel values. The partially ripe strawberry (strawberry-3) had a wide hue distribution from 22.44±6.31, while the unripe strawberry (strawberry-4) exhibited a hue is 30.08±4.35. As with the standard color tests, we found that the saturation and value measurements were inconclusive because of constant lighting conditions, whereas the hue channel remained reliable for differentiation.

In Fig. 4c, we present the real-time 2D shape and size estimation of various objects using a collocated camera and distance sensor. We tested standard objects—a cube, sphere, and pyramid of different sizes (3, 4, and 5 cm)—as well as strawberries of various shapes and sizes. The size refers to the characteristic size for each object (diameter for a sphere, edge length for a cube, edge length for a pyramid). See EM section 4.6 for a detailed description of the setup and methodology. Each object was placed at distances of 8, 10, or 12 cm from the camera, and its 2D shape was identified as a quadrilateral, circle, triangle, or random shape, depending on its actual form. Our results demonstrate a ~100% success rate with 95% Wilson confidence intervals (CI) of [0.57,1.00], where n=5 for shape detection at all tested distances (see Supplementary Fig. 7).

We also measured the size of these objects with high accuracy. Fig. 4d shows the measured perimeters for the standard objects and three strawberries. Among these, 3 cm diameter circular objects and strawberries showed the highest mean absolute percentage error (MAPE), at 8% and 10%, respectively. However, the MAPE decreased for larger objects, dropping below 1.8% for cubes at 8 cm (see Fig. 4e). We provide all estimated shape and size data in Supplementary Figs. 7–11, including both masked and real images of the objects.

2.4. Curvature sensor
Curvature sensors are a type of proprioceptive sensor that combines pressure data and bending data. These sensors serve two main purposes: (1) providing bending state information that complement tactile measurements, and (2) enabling load estimation and slip detection. However, the primary function is to monitor the bending state of each finger during grasping. This feedback is essential because it is an underactuated system where all fingers are actuated simultaneously. Curvature sensing allows us to confirm that each finger bends sufficiently to establish stable contact and apply the required force without damaging the fruit.

In Fig. 5a, we present a fully integrated optical fiber sensor embedded in a gripper’s finger, along with an image illustrating red light traveling through the optical fiber from the transmission end (red LED) to the receiver (phototransistor). We designed the finger actuator to function as a curvature sensor. We created a U-shaped channel measuring 0.8 mm in diameter on the palmar side of the finger cover and installed 23 soft clamps to secure the fiber along this channel. The sensor design and characterization procedures are described in EM section 4.7 and SI section 6.

In Fig. 5b, we report the total light received at the phototransistor's end, expressed as voltage output over time at various bending angles. Fig. 5c displays a superimposed image of the gripper with the integrated light guide, highlighting its corresponding bending angles at different applied pressures. We converted the voltage output data into light intensity loss (dB) using Beer-Lambert’s Law, as represented by the following equation22:
Light Intensity Loss (dB) = 10×log 10 (V0/V)

The curvature sensor results indicate that light intensity initially increased from 3.58 dB to 5.09 dB as the bending angle increased from 0° to 77.47°. The bending, however, continued from 77.47° to 211°, and the light intensity dropped to -7 dB, suggesting a critical bending angle for sensor measurements. Fig. 5d plots bending angles against their corresponding light intensity variations at different input pressures. The initial increase in light loss can be explained by our sensor assembly process. While integrating the fibers into the finger, there is micro-bending between the fibers and the clamps, due to minimal compressive loading. This slight bending produces a characteristic response during finger bending at small bends, the stress relaxes, the bends straighten, and the optical signal changes gradually, whereas beyond a critical angle (~75–80°) the fibers enter tensile stretch, and the light intensity decreases sharply. However, the raw data are non-monotonic, which can be compensated by normalizing the light-loss signal with the gripper’s internal pressure (Fig. 5d). This normalization transforms the response into a monotonic relationship that can be directly interpreted as bending angle versus normalized light loss (see Supplementary Fig. 12). Additionally, we demonstrated that the curvature sensor can be used for load capacity estimation and slip detection. We integrated this sensor into the gripper and tested it by holding various loads (200 g, 400 g, and 600 g) at different pressures (70, 80, and 90 kPa). The results showed that the light intensity dropped by 50% when holding 600 g compared to 200 g, effectively capturing load variations. Furthermore, the sensor successfully detected instances of weight slipping, demonstrating its capability to not only sense bending but also detect grasping failures (Fig. 5e). This capability also allowed detection of slipping events during heavy or unstable grasps. While the present study focuses on demonstrating integrated sensing, we envision applying this capability in future automated harvesting systems to sort fruit into weight categories (light, average, heavy).

2.5. Tactile sensor
In Fig. 6a, we present the tactile sensor design, showing top, side, front, and isometric views. Details regarding the sensor design and fabrication are provided in the corresponding section. Fig. 6b illustrates the application of the tactile sensor as a wearable device on a fingertip, where signals travel from the source to the receiver. In this figure, the red component represents the curvature sensor, while the blue component corresponds to the tactile feedback mechanism.

Additionally, we depict the experimental setup and the tested objects used to evaluate the sensor, covering both side and top contact interactions (see Fig. 6c and Supplementary Figs. 13a–b). Specifically, Fig. 6c and Supplementary Fig. 13a show the sensor tested under top-contact conditions with materials of varying stiffness, including extra soft (𝐸 = 55 kPa, shore 00 - 10), soft (𝐸 = 152 kPa, shore 10A), and rigid (𝐸 = 1.7 GPa, shore 80D), as well as a strawberry. The experimental setup and methods are described in EM section 4.8. The side-contact setup is shown in Supplementary Fig. 13b.

To assess the sensor’s performance, we developed five sensors (S1–S5), one for each finger of the gripper, and characterized each before final assembly. In Fig. 6d, we present the force and displacement data for sensor S1 in both side- and top-contact conditions while interacting with various objects. The results indicate that the rigid object led to the highest force (approximately 4 N) at a displacement of 0.7 mm. Similarly, Fig. 6e shows the temporal power loss data for the same force and displacement experiment. As expected, the highest light loss (-8.91 dB) occurred when the sensor interacted with the rigid object, while the lowest light loss (-4.23 dB) was observed with the extra soft object. In Fig. 6f and Supplementary Figs. 13c–d, we present the force and displacement data against light intensity loss to estimate the tactile response. Furthermore, the side- and top-contact characterization data was highly repeatable, with no notable differences between the two conditions.

The force, displacement, and light intensity loss data for sensor S2 are shown in Supplementary Figs. 14a–h. For a displacement of 0.7 mm, the maximum and minimum force and power loss values in the side-contact condition were 4.18 N and -8.77 dB, and 0.5 N and -5.29 dB, respectively. In top-contact conditions, the corresponding values were 2.28 N and -6.20 dB, and 0.27 N and -0.45 dB.

Similarly, Supplementary Fig. 15a–h presents the data for sensor S3. At a displacement of 0.7 mm, the maximum and minimum force and power loss values in the side-contact condition were 4.61 N and -9.12 dB, and 0.52 N and -5.20 dB, respectively. In the top-contact condition, the corresponding values were 1.80 N and -6.13 dB, and 0.35 N and -1.3 dB.

For sensor S4, the results are provided in Supplementary Fig. 16a–h. At a displacement of 0.7 mm, the maximum and minimum force and power loss values in the side-contact condition were 3.86 N and -8.68 dB, and 0.6 N and -5.07 dB, respectively. In the top-contact condition, the corresponding values were 4.28 N and -8.73 dB, and 0.3 N and -3.75 dB.

The data for sensor S5 are shown in Supplementary Figs. 17a–h. At a displacement of 0.7 mm, the maximum and minimum force and power loss values in the side-contact condition were 4.06 N and -9.01 dB, and 0.47 N and -3.39 dB, respectively. In top-contact conditions, the corresponding values were 3.44 N and -7.21 dB, and 0.33 N and -2.65 dB.

The tactile sensor exhibited consistent performance across all tests, showing a maximum light loss of -9 dB when compressed against a rigid object at a constant displacement of 0.7 mm with an applied force of approximately 5 N (see Fig. 6f). Furthermore, both top and side-contact experiments yielded similar light loss results, demonstrating the sensor’s ability to effectively distinguish between a rigid object (𝐸 = 1.7 GPa, shore 80D), an extra soft object (𝐸 = 55kPa, shore 10A), and a strawberry.

Fig. 6g presents a bar chart summarizing stiffness, displacement, and light loss data for all sensors, categorized by material stiffness. Additionally, we tested sensor S1 on three different ripe strawberries (Str1, Str2, and Str3). The force and displacement data indicate that the maximum force and light loss were recorded for Str2, with values of 1.36 N and -2.51 dB, respectively. The minimum force and light loss were observed for Str3, measuring 0.56 N and -1.30 dB. The side- and top-contact data for each strawberry were highly consistent, with negligible differences. All related data are reported in Supplementary Fig. 18a–f. Details of the experimental design are discussed in EM section 4.8.

2.6. Integrated gripper demonstrations
Fig. 7a-b shows the fully assembled gripper mounted on the robot arm (see Supplementary Movie 1 for the rotational module demonstration). We secured the control input, including the Raspberry Pi and PCB, to the rotational module using screws and nuts. We mounted the fully assembled gripper onto the end effector of the MyCobot 320 robot arm (Elephant Robotics, 6 DOF, 1 Kg payload) using a custom 3D-printed mount. We fixed one section of the mount to the robot arm and integrated the other section into the rotational module, securing them together with screws. Supplementary Figs. 19–22 show the detailed assembly and circuit designs. We also describe in detail the control interfaces, circuit components, and their fabrication in EM section 4.9 and SI section 7.

We developed independent control algorithms for the gripper and the robot arm. Using the Robot Operating System (ROS) on a Linux platform, we programmed the arm by manually drag and teach it to the desired waypoints and then automated its movements with ROS scripts. We controlled the gripper with a Python-based algorithm on a Raspberry Pi (Thonny interface), sequencing the activation of the vision sensor, measurement of tactile and curvature responses, object grasping, harvesting and releasing. To coordinate both systems, we manually triggered the gripper’s control code when the robot reached each waypoint. We synchronized their operations by adjusting delays between the robot’s movements, ensuring the gripper captured images and grasped the object precisely upon contact After this step, we manually initiated the harvesting module (see Table 1 and Supplementary Tables 1-3). Furthermore, a complete flow diagram of robot grasping strategies is shown in Supplementary Fig. 23, and the proposed control method allows the robot to semi-autonomously identify objects, grasp them, and characterize their tactile response, see EM section 4.10 and SI section 8. We tested various objects for real-time manipulation, including rigid and soft hemispheres, avocados, and strawberries. We describe the testing procedures in EM section 4.11.

In Fig. 7c and Supplementary Fig. 24, we demonstrate real-time grasping of both rigid and ultra-soft hemispherical objects. The object shapes were accurately classified as circular, with perimeter measurements yielding MAPE value of 5.9% and 7.6%. The hue distributions of both objects were 88.39±8.25 and 93±7.66 consistent with the black-colored cube used in earlier trials as a reference (see Supplementary Fig. 5, Supplementary Fig. 24a-b, and Supplementary Fig. 24e-f).

We also reported tactile and curvature sensor measurements in Supplementary Fig. 24c-d and Supplementary Fig. 24g-h, respectively. The results showed that at least one finger typically recorded the highest light loss in both sensing modalities. For the rigid object, finger 5 exhibited the maximum tactile light loss of −6.56 dB, while the highest curvature change was −5.5 dB. In contrast, for the soft object, finger 2 recorded the greatest tactile response at −1.92 dB and registered the maximum curvature change of −7.09 dB. See Supplementary Movie 10 real-time demonstration.

We further evaluated the gripper’s performance by grasping ripe strawberries purchased from a supermarket under both light and dark conditions. The gripper successfully grasped the strawberries without bruising them and measured their color, size, and tactile response (see Supplementary Fig. 25a-d). The strawberries were positioned on a scissor lift and grasped from above (see Fig. 7d and Supplementary Fig. 25a). As before, one finger showed the highest tactile signal: finger 2 recorded a maximum light loss of −2.22 dB, while the maximum curvature changes of −7.70 dB (see Supplementary Fig. 25c-d). The measured hue value was 171±24.53, and the shape estimation yielded a MAPE of 9.75% (see Supplementary Fig. 26b). All individual finger measurements are reported in Supplementary Fig. 25c-d and demonstrated in Supplementary Movie 11.

The gripper was also tested on ripe and unripe avocados. For unripe avocados, the shape was correctly identified as circular, with a hue distribution ranging from 64.58±14.44, consistent with earlier cube measurements (see Supplementary Fig. 26a-b). Finger 1 recorded the highest tactile light loss of −7.5 dB, while the maximum curvature recorded was −8.28 dB (see Supplementary Fig. 26c-d). For ripe avocados, the gripper also predicted a circular shape, with a hue distribution of 102.29±47.47, accurately capturing the color transition from green to dark purple (see Supplementary Fig. 26e-f). Finger 2 recorded the maximum tactile light loss (−3.75 dB) and curvature response (−8.85 dB) (see Supplementary Fig. 26g-h). These results confirm that the gripper successfully distinguished between ripe and unripe avocados using both color and tactile feedback.

Finally, we demonstrated real-time strawberry picking from a live strawberry plant (Fig. 7e-f, Fig. 27, and Supplementary Movie 12). The gripper successfully identified and grasped both ripe and unripe strawberries, and measured their shape, size, color, and tactile responses. For ripe strawberries, finger 4 recorded the highest tactile light loss (−1.08 dB) and curvature change (−5.08 dB) (see Supplementary Fig. 27c-d). For unripe strawberries, finger 4 also showed the strongest response: −3.59 dB in tactile and −4.58 dB in curvature (see Supplementary Fig. 27g-h). The hue values were 135.29±63.43 for ripe and 29.59±5.33 for unripe strawberries (see Supplementary Fig. 27a and Supplementary Fig. 27e). Shape estimation yielded MAPE errors of 13.67% for ripe and 1.8% for unripe strawberries (see Supplementary Fig. 27b, Supplementary Fig. 27f, and Supplementary Movie 12). Finally, we also demonstrated the successful picking of a ripe strawberry using the rotational module without bruising the fruit (see Supplementary Fig. 27a-h, Supplementary Fig. 28a-c, and Supplementary Movies 13–14). Furthermore, we present a compiled plot of tactile measurements across all tested objects to highlight the clear differences in tactile response between them, as shown in Fig. 7g.

Methods
4.1. Gripper design and fabrication
The dimensions of gripper components are as follows: the finger actuator has bases measuring 21.85 mm and 10 mm, legs of 52.83 mm, and a height of 12 mm. The finger holder has a pentagonal shape, with each side measuring 22.35 mm and a height of 20.75 mm. Each face of the holder features a rectangular cavity measuring 11 mm by 8.95 mm by 5.65 mm, with an inlet tube for attaching the actuators. The holder also includes a slot for a camera and a ToF sensor, along with additional slots for the laser diode and phototransistor mounts (Fig. 1a and Supplementary Fig. 1).

To fabricate the gripper, we used a multi-step process. First, we designed 3D models of the finger actuators, the holder, and other components in SolidWorks 2022 (SolidWorks corps). To simplify printing, each finger actuator was split into two parts: a finger cover and a finger body. In contrast, the finger holder was designed as a single piece. Next, we printed the components on a high-resolution DLS printer (Carbon M1, Carbon Inc.). Finally, we assembled the printed parts using thermal baking (see SI Section 1 for details) and fastening methods such as adhesive bonding, screws, and nuts.

The gripper construction follows a multi-material approach, combining both soft and rigid components. Each finger is made of a soft silicone-polyurethane elastomer (SIL 30, 𝐸 = 1 MPa, 𝛾 = 3.50). The finger holder chassis, tube connectors, camera, ToF sensor, and laser diode 𝐸 =1.7 𝛾 =0.3). Details of the printing protocols are provided in SI section 1 and Supplementary Fig. 2.

4.2. Rotational module design and fabrication
The rotational module relies on gripper rotation about its roll axis. However, this axial rotation also twists the sensor wiring, power supply cables, and pneumatic tubes. To prevent these twists, we divided the rotational module into two parts: a fixed part attached to the gripper end effector and a rotating part that connects the gripper to the control unit. We used planetary gear-based rotational joints to link these parts. We also created a hollow central axis and integrated a concentric slip ring, allowing wires and the pneumatic tube to pass through freely, ensuring a secure connection between the gripper and control unit without twisting.

A high-torque DC gear motor powers the gear transmission, which consists of a spur gear and a pinion. The pinion is fixed to the rotating section, while a floating spur gear connects to the DC motor. We incorporated ball bearings to enable smooth rotation and minimize eccentricity45. We mounted the control unit on the outer perimeter of the rotating part and controlled the DC motor with a double-pole double-throw (DPDT) switch (Supplementary Fig. 1 and Supplementary Movie 1). We used the same 3D printer to fabricate most of the rotational and fixed sections, along with the gears, connectors, and bolts. We printed these components using rigid polyurethane, the same material used for the finger holder. For assembly, we employed a traditional method using screws and bolts. The detailed printing protocol is discussed in SI section 1.

4.3. Finger bending characterization
We printed a finger actuator with an integrated rigid body to house the laser diode, phototransistor, and pneumatic tube connectors (see Fig. 2d). We then secured the rigid body to a table using a bench vise. To track the bending angle, we placed grid paper behind the actuator and marked the finger with permanent markers. We positioned a tripod-mounted mobile phone camera (with at least 10 megapixels (MP) resolution) at a fixed distance from the finger actuator, ensuring consistent focus and preventing any camera movement during the experiment. We measured the bending angle of the finger actuator at pressure inputs ranging from 0.1 to 1 bar, in increments of 0.1 bar. We used a constant air source with a pressure regulator to maintain a stable air supply. For each pressure setting, we held the pressure for 60 seconds to achieve quasi-static conditions before taking a photograph of the actuator. Afterward, we imported the images into ImageJ, an open-source software, to determine the relationship between bending angle and pressure. We present all results in Section 2.

4.4. Finite element method simulation
We used Ansys 2022 software to conduct finite element simulations and assess the mechanical feasibility of a soft robotic gripper. Our simulations analyzed bending angles, blocking force, and overall gripper response. We examined total displacement (mm), total strain (mm/mm), and equivalent von Mises stress (MPa) for both individual fingers and the entire gripper. To develop the simulation model, we imported SolidWorks files into Ansys Workbench 20.1 and applied the Static Structural module. We added custom material properties to the Ansys material library using elastomer stress-strain data obtained from standard tensile testing (ASTM D412, Die C, 500 mm/min). After fitting the experimental data into a hyperelastic model, we found that the Yeoh 3rd order model provided the best fit, with the following coefficients: C10 = 167,499.9 Pa, C20 = 453.53 Pa, and C30 = 5.5374 Pa. For meshing, we applied a 1 mm nonlinear quadratic mesh, generating 54,514 nodes and 32,908 elements. We set gravity as a body force in all simulations. To simulate the blocking force, we introduced a fixed surface to measure the actuator’s force output. We set contact stiffness for both self-contact and the fixed surface to 200 N/mm³ using the pure penalty method, allowing a penetration of approximately 0.1 µm. We performed nonlinear simulations using the Newton-Raphson method on a 12th Gen Intel(R) Core™ i7-12650H CPU with 64 GB RAM. The bending angle simulation took 17 minutes and 38 seconds, requiring 298 iterations to converge, while the blocking force simulation took 36 minutes and 7 seconds, completing 738 iterations.

4.5. Vision sensor design and fabrication
The vision sensor consists of an RGB camera and a ToF sensor system, built in a collocated configuration. We built a sensor mount that attaches to the center of the palmar side of the finger holder, positioning the sensors side by side in the center of the gripper. These sensors are securely fixed and move with the gripper. For vision, we used a commercial Raspberry Pi camera with a 5-megapixel Arducam 1/4-inch Omnivision OV5647 sensor, featuring a fixed-focus lens, a 160° field of view, and a frame rate of 30 fps, suitable for Raspberry Pi 4 B. For depth sensing, we integrated a ToF Laser Range Finder with a 940 nm laser detector (VL53L0X) capable of measuring distances up to 2 meters. Both sensors were purchased from Amazon.com. To integrate these sensors, we designed a custom slot that holds them securely through a tight fit (see Supplementary Fig. 4). We fixed the entire mount, including the sensors, using through-holes in the finger holder and double-sided tape.

4.6. Vision sensor test
To test the vision sensor, we built a dedicated setup. First, we mounted both the camera and the ToF sensor on the soft gripper to replicate real testing conditions, then attached the gripper to a 3D-printed stand (PLA, Ultimaker 4). We 3D-printed three standard shapes—a cube, a sphere, and a pyramid—and used three different strawberries. To ensure consistent lighting, we used an Amazon studio box with white light (~832 Lux) and a scissor lift stand to adjust the height of the objects for vision measurements. We used Dr. Meter Lx11330B for light measurements. See Fig. 4a and Supplementary Fig. 4 for an overview of the setup.

Color: We used cubes (side length = 5 cm) and attached stickers with six different colors (white, black, red, blue, yellow, and green) to evaluate the camera’s color detection algorithm. Each cube was placed on a scissor lift at a fixed distance of 8 cm from the camera, and measurements were taken (Supplementary Table 1). For color detection, we recorded Hue, Saturation, and Value (HSV) data and plotted them in a histogram with probability mass functions. The Hue range was 0 to 180, while Saturation and Value ranged from 0 to 255. We repeated this procedure for all six colors, and the results are presented in Fig. 4b-e and Supplementary Figs. 5-7.

Size estimation: For the size estimation test, we used the same vision sensor setup as in the color detection test. We tested 12 distinct objects with four different shapes: a triangular pyramid, a sphere, a cube, and a strawberry. The triangular pyramids had three different edge lengths (3 cm, 4 cm, and 5 cm), the spheres had three different diameters (3 cm, 4 cm, and 5 cm), and the cubes had three different side lengths (3 cm, 4 cm, and 5 cm). The strawberries varied in size. Each object was placed on a scissor lift, and its outer perimeter was estimated using image data and ToF sensor readings at three distances (8 cm, 10 cm, and 12 cm). To estimate the true object size, we required two pieces of information: (1) the camera’s focal length and (2) the object’s perimeter in the image domain. We calibrated the camera under a pinhole model by placing a cube of known size at various distances and then applied the pinhole method with similar triangle techniques to calculate the actual object size:
f = (dk * v) / h
Where is f the focal length, d is the distance of the object along the principal axis, v is the number of vertical pixels on the square target object, h is the true height of the object.

To ensure the accuracy of ToF sensor measurements, we calibrated the sensor by comparing its readings with objects at known distances. We used this data to develop a calibration curve that mapped the ToF sensor measurements to actual distances and then integrated it into the size estimation code. Additionally, we ensured that the camera and ToF sensor were properly aligned so their coordinate frames coincided (i.e., the camera’s principal axis was aligned with the ToF sensor). This alignment guaranteed that the ToF sensor provided the correct distance between the camera and the object for every measurement.

We also applied temporal smoothing to improve size estimation accuracy across multiple frames. We used a receding horizon filter with a window size of 20 frames, pushing each new size estimate into a queue and dropping the oldest estimate. Several consistency checks (e.g., a shape test and a size percentage error test) were performed before adding any data point to the queue. The output size estimate was then calculated as the average of the 20-frame window (Supplementary Table 2).

Shape estimation: For the shape estimation, we kept the same setup as the previous experiments, except that the objects’ sizes were kept at 4cm. We used triangular pyramids, spheres, cubes, and strawberries for the shape estimate. We only measured the 2D geometry for shape detection, so triangular pyramids were classified as triangles, spheres were classified as circles, and cubes were classified as quadrilaterals (Supplementary Table 3). Strawberries were classified as unknown. The results are presented in Fig. 4b-e and Supplementary Figs. 5-7.

4.7. Tactile and curvature sensor design and fabrication
We used a 0.7 mm stretchable, clear thermoplastic polyurethane (TPU) fiber from Crystal Tec Korea (purchased via Amazon). We connected both ends of the fiber to the laser diode and phototransistor mounts. Each mount is cylindrical, with an outer diameter of 5.5 mm, an inner diameter of 3.1 mm, and a 0.8 mm fiber holder. We placed and glued these mounts into slots on the finger holder. Furthermore, for the light emitter, we employed a 3 mm laser infrared diode (1.5 mW, 850 nm), and for the receiver, we used a phototransistor with a spectral response of 450–1080 nm (purchased from digkey.com). When the finger bends, it deforms the fiber, that changes the light intensity. We measure this light intensity variation as a voltage output.

We designed the tactile sensor as a modular, U-shaped conduit—2 mm thick and 10 mm wide—composed of both soft and rigid materials (Fig. 6a). The base conduit is rigid and houses the diode and phototransistor mounts, as well as the fiber. The second layer (the contact layer) is soft and features a 0.8 mm diameter channel with 19 integrated soft clamps along the conduit. We used the same mount dimensions and light components (laser diode and phototransistor) as in the curvature sensors (Fig. 6a).

To fabricate the tactile sensors, we first 3D-printed a rigid polyurethane structure (RPU70) measuring 1.25 mm in thickness, followed by a 0.75 mm silicone urethane layer (SIL30) containing the channel and clamps. We joined these layers using wet resin and then baked them in an oven (see SI section 1 for printing procedure details). Next, we threaded the TPU fiber through the conduit channel, installed the fibers, laser diode, and phototransistor, and finally mounted the completed sensor onto the fingers. We used Black Panther™ 85 (𝐸 = 4.83 MPa, 𝛾 = 1.21) as a detachable adhesive to attach the tactile sensors to the fingertips (Fig. 6b).

The optical fibers for both curvature and tactile sensor are bonded into rigid holders, and the holders are bonded to the fingers, so the relative geometry is fixed during operation. The light source and photodetector (LED and phototransistor) are rigidly mounted to the holder and encapsulated with an opaque silicone sealant, that blocks stray light and further stiffens the source–fiber and fiber–detector interfaces against vibration or shift. Additionally, emitting/detecting elements are enclosed and spectrally separated from typical ambient illumination, the sensor output is largely insensitive to room lighting.

4.8. Tactile sensor characterization
We built and characterized five tactile sensors before integrating them into the gripper. EM section 4.1 provides further details on the design and fabrication of these sensors. We powered each sensor’s laser diode and phototransistor using the same circuit from the curvature sensor tests and recorded the phototransistor’s voltage output through an Arduino Uno. Because the sensor is U-shaped, it offers two contact faces (top and side), so we tested both faces independently for each sensor.

We interpret the sensor’s output as a voltage change, which corresponds to deformation and light intensity loss. However, effective tactile feedback requires both force and displacement information, so we also correlated the force output with the voltage change at a given displacement. To perform this task, we used a micro universal testing machine (maximum 10 N load cell; Cell Scale) to apply a compressive load on the sensor while measuring force, displacement, and voltage. We attached each sensor to the machine's moving shaft with a custom 3D-printed fixture that allows both orientations of the sensor to be tested. We ensured that each sensor face and the object remained perfectly parallel (see Supplementary Fig. 13a-b for the setup).

We tested the sensors against four types of materials: an extra soft material (Ecoflex 10, Shore 00-10, E = 55.2 kPa at =1.0), a soft material (Dragon Skin 10, Shore 10A, 𝐸 = 152 kPa, 𝛾 = 1.0), a rigid material (RPU 70, Shore 80D, 𝐸 = 1.7 GPa, 𝛾 = 0.3), and actual strawberries. We molded the extra soft and soft materials into semispherical shapes and 3D-printed a rigid semispherical shape in PLA (Ultimaker). For each sensor face, we zeroed the load cell, positioned the sensor to barely touch the object, and applied a 0.7 mm displacement under displacement control. We ran seven compression tests per material, sampling force, displacement, and voltage data at 10 Hz. We also captured images at 1 Hz using the testing machine’s integrated camera. We repeated these tests for both faces of all five sensors and presented the results in section 2 and Fig. 6e-g, Supplementary Fig. 13c-d, Supplementary Figs. 14-18.

4.9. Control unit design and fabrication
We designed the gripper’s control unit around a Raspberry Pi to manage inputs from 13 sensors and two solenoid valves. The 13 sensors include one camera, five analog tactile sensors, five analog curvature sensors, one analog pressure sensor, and one digital ToF sensor that communicates via the I²C protocol. We used two digital outputs to control the valves, which brought the total I/O signals to over 70 wires. To handle these connections efficiently, we developed a custom printed circuit board (PCB) that interfaces the gripper’s I/O signals with the Raspberry Pi.

Because the Raspberry Pi only supports digital communication protocols (I²C, SPI, and UART), we used three high-resolution 16-bit analog-to-digital converters (Adafruit ADS1115) to read the 11 analog sensors and transmit their data to the Raspberry Pi via I²C. We also added an I²C port on the PCB for the ToF sensor. To drive the valves, we used two Raspberry Pi GPIO pins connected to a motor driver (1.2 A, 12 V Pololu 8833), which we integrated into the PCB alongside the two solenoid valves. Finally, we linked the PCB to the Raspberry Pi with jumper wires.

The valves require more current than the Raspberry Pi can supply, so we powered them from an external source (bench power supply). We also used an external uninterruptible power supply (UPS, 5 V output, 1A max) for the Raspberry Pi. We routed power from both the UPS and the external supply through a slip ring before delivering it to the Pi and solenoid valves. To accommodate this, we modified a USB-C cable for the Raspberry Pi connection. Supplementary Figs. 19-23 present the complete diagram and circuit design.

4.10. Robot arm control code
For the robot arm, we used the M320 Cobot robot. We took advantage of its built-in drag-and-teach functionality to write a Python script that allowed us to plan the robot’s trajectory. The drag-and-teach feature lets you program a path by physically moving the robot through a series of waypoints. After starting this script in the command prompt, we moved the robot through the desired set of waypoints using manual control by human operators, which were recorded and saved in a JSON file. Once the waypoints were created, we wrote another script to iterate through them, directing the robot to move to each waypoint, pause for a set period, and then continue to the next waypoint. This script would loop indefinitely unless it was halted, ensuring the robot would continuously execute this path (see Supplementary Fig. 23).

Using these scripts, we programmed the four-step path that the robot would follow during the integrated test (not picking fruits directly on the plant). Each test began with the robot in its default position, with the arm extended straight up (State A). From there, the robot moved to a position where the gripper’s camera faced directly downward over the object (State B). Next, the robot moved straight down so the gripper contacted the object (State C). Finally, the robot moved to an intermediate position away from the object (State D). The robot repeated this sequence from State A through State D for each test (see Table 1: Algorithm 1).

Gripper Control Code: The onboard Raspberry Pi was responsible for collecting all the vision and tactile data from the gripper. We developed a single Python script that handled all data collection. The script consists of two main parts: the vision code and the grasping code.

Vision Code: The vision code was the first part of the Python script, running whenever the script was called. It activated the camera and used a computer vision algorithm (OpenCV) to detect objects within the camera’s field of view (FOV). See Supplementary Tables 1-3: Algorithms 2-4 for a detailed description of the algorithm. The script saved an RGB image of the detected object with a bounding box around it, separate HSV histogram files in image format, and a segmented image of the object printed with shape and size information.

Grasping Code: The gripper control code was the second part of the Python script, running immediately after the vision code finished. It collected data from five curvatures and five tactile sensors (as voltage outputs), as well as the gripper’s pressure and the time elapsed. Once activated, it triggered the solenoids to allow air from the air compressor into the gripper. As the gripper inflated, the onboard pressure sensor monitored the pressure. When the gripper reached the desired pressure, the solenoids were closed, maintaining the pressure without releasing air.

4.11. Integrated robot arm and gripper test
For running the integrated tests, we ensure consistent room lighting conditions (~421 lux) for all the testing operations. The integrated test required two separate codes (the gripper control code and the robot arm control code), described in EM section 4.10. We built these codes separately on different devices. The gripper control code ran on the onboard Raspberry Pi, while the robot arm control code ran on a separate computer with a Linux operating system. In the experiment, we ran these codes sequentially, starting with the robot arm control code and then running the gripper control code.

The integrated test includes full-scale robot arm manipulation and real-time measurements of various objects. The robot manipulation task consists of moving the robot close to the object, taking vision measurements of the object, grasping the object with the soft gripper, obtaining tactile and curvature measurements, and then lifting the object and moving it to a different location. We selected various objects to demonstrate the gripper’s manipulation capabilities. Initially, we used synthetic objects such as 3D-printed rigid and soft (casted) hemisphere objects. After confirming the gripper’s performance with synthetic objects, we tested the gripper on strawberries and avocados (purchased from the supermarket). Finally, we tested real fruit picking on a strawberry plant grown in a greenhouse using the gripper’s rotational module.

For the integrated test, we mounted the robot arm on a workbench using clamps. We placed the objects on a separate table next to the workbench with a brown-colored background to create a sharp contrast between the objects and the background. We placed the object on a scissor lift so we could adjust its height and ensure it remained within the robot arm’s workspace. We used an air compressor to supply air to the gripper and a bench power supply to power the solenoids. Additionally, we used an uninterruptible power supply to power the Raspberry Pi. See Supplementary Fig. 22 for an image of the setup.

For the real plant setup, we used a similar arrangement, except the fruits were hanging on the plant, and we programmed a custom path so the gripper would move to the plant and grasp the fruit. See Supplementary Fig. 28 for an image of this setup.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This manuscript presents perhaps the most advanced/integrated soft robot harvesting solution to date, combining state-of-the-art technologies in terms of actuation, proprioceptive sensing, tactile sensing, and computer vision. Although the technologies themselves are not new, the combination of all of them in a single system puts the authors in a unique position to investigate how the technologies can be integrated in a meaningful way for harvesting fruits. This would be a major contribution to the field, where much research work has focused on a single technological component only.\nHowever, the reviewer feels the manuscript only partially addresses this question. Although the hardware is integrated, the characterisation (e.g. of the actuator bending angle), calibration (e.g. the curvature sensing), and control is still focused on the individual components.\n- Figure S22 gives a hint on how all technologies can be integrated/deployed in a meaningful way, but it feels incomplete:\n-- First, it is for grasping rather than harvesting, as the activation of the rotational harvesting module is not included.\n-- Second, the data from the proprioceptive sensors is not used, raising questions about the necessity of these sensors. How does knowing the curvature of the actuators contribute to harvest the strawberry? The authors mention that the proprioceptive sensors can be used to estimate the load and detect slipping, which sound more meaningful for harvesting applications. However, the existing calibration for weight is not convincing, as it does not report whether the ability to detect different weights still holds when the objects are of different sizes. Is it still possible to detect weight when the object size (and thus gripper curvature) varies? What is the accuracy and resolution of the load detection capability (e.g. can it be used to make a distinction between light, average, and heavy strawberries for sorting purposes)?\n-- Third, the description of how the tactile data is used is unclear. In 4.10 it is stated that \"As the gripper inflated, the onboard pressure sensor monitored the pressure. When the gripper reached the desired pressure, the solenoids were closed, maintaining the pressure without releasing air\", suggesting closed loop control by incrementally increasing the air pressure applied to the gripper until a desired grasping force is achieved, ensuring gentle grasping without damage for strawberries of different sizes. However, it is not clear what the 'desired pressure\" is and whether pressure refers to air pressure or tactile sensor pressure. Moreover, in Table 1, this closed-loop control is not visible. In Fig S22, there is a step named \"Input pressure = threshold\", but again it is unclear if it is talking about air pressure or tactile sensor pressure and the threshold is not quantified.\n-- Fourth, did the authors explore usage of the tactile sensor data beyond optimising the applied air pressure to the gripper? (e.g. measure ripeness to decide whether to harvest yes/no?, or measure ripeness to decide whether to release in container A, B, or C?, or detect slipping during the rotational harvesting and increase the air pressure applied to the gripper if this occurs?)\n-- Fifth, for the ripeness detection, it feels as a missed opportunity to combine the ripeness estimation from the vision system with the ripeness estimation from the tactile sensors to get a more accurate estimate. Again, it would be good to incorporate this in the flowchart.\nSummarising, it would be good to have a flowchart for the harvesting task where it becomes clear how all of the actuators and sensors are linked and together contribute to successful task execution.\n- Several design decisions are not clear. Why did the authors opt for 5 fingers? Why the tapered shape? Why are the fingers aligned in the same plane? Please explain how this choice of integration was found the most suitable for the harvesting task.\n- Regarding the fibre sensors, Fig 5(a & c) show that the fibre makes a sharp bend before entering the light source and sensor. Could this explain the initial increase in light intensity (i.e. that initial bending of the actuator allows the fibre to straighten a bit around those sharp bends)?\n- The sensor system may be prone to drifting. How constant were the sensors values over time (e.g. over 3 hours of continuous use or separate experiments one week apart)? How did the authors address this issue\n- In Figure 7 and the supplementary videos, the bellow segments appear to have a black marking/cover that is not visible in figure 5 and 6. Does this serve a purpose?\n- The legend of Fig.6 (e) shows six different data series, but only three colours can be observed in the plot. Do they fully overlap?\n- Please check the paper for some small typos (e.g. Table 1--> \"volatge, \"clossed\", \"valtage\", \"distnace\", line 233 \"both had hues are\")\n- Have figure S6 row 1, middle and right image been accidentally swapped?\nOverall, this research work has potential for being published in Nature Communications, but the manuscript feels incomplete at this stage. A stronger linking between the sensors and actuators would greatly enhance the impact and contribution of the manuscript.", "reviewer_2": "The reviewed work addresses an important challenge in robotics -- soft visuo-tactile manipulation. Physical intelligence through soft robotics that you are tackling is right on time.\nThe comments are attached.", "reviewer_3": "The authors present a soft pneumatic gripper with integrated proprioceptive and tactile sensing. By combining these capabilities with vision, the gripper can assess fruit ripeness and perform grasping/picking tasks without causing damage. Thorough characterization of the gripper has been conducted and its potential application in fruit picking has been demonstrated.\nOverall, the paper is well-written and organized. However, the following issues need to be addressed before it can be considered for publication in Nature Communications:\n1. The comparison with related work is insufficient. The authors claim that the proposed method (collocated camera and ToF sensor + tactile sensing + proprioception) is better than traditional methods (camera + tactile sensing), e.g., for handling occlusions, but the strawberry picking demonstration does not substantiate this advantage over existing techniques. A more detailed comparative analysis, backed by stronger experimental evidence, is required to highlight the method’s novelty and justify its claims.\n2. The light intensity loss curve in Fig. 5d is non-monotonous, which creates ambiguity, as a single intensity value could correspond to two different bending angles. The authors should discuss how they resolve this ambiguity in practice. Furthermore, during grasping, the curvature sensor may come into contact with the fruit. It is unclear how this unmodeled physical contact would affect the sensor’s accuracy, and a discussion on the curvature sensor’s robustness to such interactions is needed.\n3. Given the fingertip sensor’s relatively large contact area, different regions of the sensor will contact the fruit during picking. How would the tactile sensor’s output (as shown in Fig. 6f) be affected by variations in the object’s shape, size, and the specific location of contact? And how would the sensor respond to unexpected shear forces, such as those from external disturbances during or after a picking motion?\n4. The optical fiber for tactile sensing is continuous across the top and side pads, yet these regions are characterized independently (Fig. 6f). Please clarify how the sensor responds when forces are applied to both the top and side simultaneously.\n5. Fig. 6d reveals a noticeable hysteresis in the sensor’s response during loading and unloading cycles. This could significantly affect the accuracy of the proposed model. Please provide a more detailed analysis of this behavior and its implications.\n6. What’s the purpose of measuring perimeters of objects (e.g., in Fig.. 4d)? Just for quantifying the accuracy of the object size estimation method, or does it contribute to the subsequent grasping/picking strategy?\n7. Please provide clear definitions for the “workspace area” in Fig. 2e and the “blocking force” in Fig. 3a.\n8. The manuscript frequently refers to object sizes, such as “a 2 cm object”, without specifying whether this refers to diameter, side length, or another dimension. This ambiguity should be resolved. Please define these terms clearly and consider annotating dimensions directly on the objects in Figs 3d or 3e. Please check and fix this issue throughout the manuscript.\n9. Fig. 3d is confusing to read at first glance. I suggest using different colors to indicate the real object from the error bars.\n10. The meaning of “80mm, 100mm, and 120mm” in Figs. 4d and 4e is unclear. Please add explicit annotations, for example, using “d=80mm” to denote diameter."}, "author_response": "Point by Point Response to the Referees\nReviewer #1 (Remarks to the Author):\nThis manuscript presents perhaps the most advanced/integrated soft robot harvesting solution to\ndate, combining state-of-the-art technologies in terms of actuation, proprioceptive sensing, tactile\nsensing, and computer vision. Although the technologies themselves are not new, the combination\nof all of them in a single system puts the authors in a unique position to investigate how the\ntechnologies can be integrated in a meaningful way for harvesting fruits. This would be a major\ncontribution to the field, where much research work has focused on a single technological\ncomponent only.\nHowever, the reviewer feels the manuscript only partially addresses this question. Although the\nhardware is integrated, the characterisation (e.g. of the actuator bending angle), calibration (e.g.\nthe curvature sensing), and control is still focused on the individual components.\nAuthors Response: We thank the reviewer for the in-depth feedback on our research article and\nappreciate the opportunity to address the points raised. As the reviewer is aware, multifunctional\nand multimodal sensing in soft robotics is a very active field of research. Matching the gripper’s\nform factor, sensorizing each finger with tactile and curvature sensors, and collocating a camera\nand distance sensor within the gripper allow us to measure the gripper’s bending state and tactile\noutput, as well as basic visual features—color, shape, and size—in real time, without concerns\nabout occlusion and with high accuracy. Since we developed this system from the component\nlevel, the initial part of the work emphasized individual component characterization to understand\nthe nature of our gripper and sensors. However, we also demonstrated the fully integrated system\nin the latter part of the work; for example, we acquired all measurements and then picked fruit (see\nFigure S24 and Video 13). We hope the remainder of our responses brings into focus our\ndemonstration of the integrated system.\nHere are the responses to specific questions the reviewer has raised.\n1. Figure S22 gives a hint on how all technologies can be integrated/deployed in a meaningful\nway, but it feels incomplete:\nAuthors Response: We thank the reviewer for this valuable feedback. We agree that the original\nFigure S22 did not fully capture the integrated picking/harvesting process. In the revised\nmanuscript, we have updated Figure S22 with a more comprehensive flow chart that clearly\nillustrates the fully integrated capabilities of our system, incorporating the reviewer’s suggestions\nto enhance clarity and completeness.\nI. First, it is for grasping rather than harvesting, as the activation of the rotational\nharvesting module is not included.\nAuthors Response: We thank the reviewer for raising this concern. We agree that the\noriginal Figure S22 did not fully capture the integrated picking/harvesting process. In the\nrevised manuscript, we have updated the flowchart to include a rotational harvesting step\nafter the system grasps the fruit. The revised figure now illustrates the complete workflow,\nfrom user input to object classification, gripper actuation, and harvesting, providing a\nclearer representation of the integrated system.\nFigure 1. Flow Chart\nII. Second, the data from the proprioceptive sensors is not used, raising questions about\nthe necessity of these sensors. How does knowing the curvature of the actuators\ncontribute to harvest the strawberry? The authors mention that the proprioceptive\nsensors can be used to estimate the load and detect slipping, which sound more\nmeaningful for harvesting applications. However, the existing calibration for weight\nis not convincing, as it does not report whether the ability to detect different weights\nstill holds when the objects are of different sizes. Is it still possible to detect weight\nwhen the object size (and thus gripper curvature) varies? What is the accuracy and\nresolution of the load detection capability (e.g. can it be used to make a distinction\nbetween light, average, and heavy strawberries for sorting purposes)?\nAuthors Response: We thank the reviewer for this valuable comment. In our design, each\nof the five fingers is equipped with a curvature and pressure sensor, which is technically\nchallenging to implement in a fully soft and underactuated gripper. Additionally, the\ngripper has a pressure sensor whose output is combined with the curvature sensor to create\nthe proprioceptive sensor.\nThe fusion of curvature and pressure serves two main purposes: (1) providing bending state\ninformation that complements tactile measurements, and from this data (2) load estimation\nand slip detection. As the reviewer notes, changes in curvature can reflect variations in\nload. For example, as shown in Figures 2c–d, tactile measurements from the gripper (Fig.\n2c) are accompanied by corresponding curvature readings (Fig. 2d) when grasping a ripe\navocado. We observed similar results with a ripe strawberry: in one case, finger 1 showed\nno curvature change, while finger 2 did, indicating differences in contact and load\ndistribution. These data are reported in full in the revised manuscript and demonstrate how\nproprioceptive sensing can identify maximum tactile force feedback (Figures S23–S26 and\naccompanying Movies 10–12), even under asymmetric grasping conditions with under-\nactuation.\nIn our experiments, curvature readings remained nearly constant up to approximately 600\ng, after which measurable changes were observed in ~100 g increments (see revised Fig.\n2a–b). This capability also allowed inference of slipping events during heavy or unstable\ngrasps. While the present study focuses on demonstrating integrated sensing, we envision\napplying this capability in future automated harvesting systems to sort fruit into weight\ncategories (light, average, heavy). We updated the role of curvature sensors in the revised\nmanuscript.\nFigure 2. Proprioception for load and curvature\nIII. Third, the description of how the tactile data is used is unclear. In 4.10 it is stated that\n\"As the gripper inflated, the onboard pressure sensor monitored the pressure. When\nthe gripper reached the desired pressure, the solenoids were closed, maintaining the\npressure without releasing air\", suggesting closed loop control by incrementally\nincreasing the air pressure applied to the gripper until a desired grasping force is\nachieved, ensuring gentle grasping without damage for strawberries of different sizes.\nHowever, it is not clear what the 'desired pressure\" is and whether pressure refers to\nair pressure or tactile sensor pressure. Moreover, in Table 1, this closed-loop control\nis not visible. In Fig S22, there is a step named \"Input pressure = threshold\", but\nagain it is unclear if it is talking about air pressure or tactile sensor pressure and the\nthreshold is not quantified.\nAuthors Response: We appreciate the reviewer’s concern. Our tactile sensor measures\nlight intensity loss caused by deformation of the embedded optical fibers under an applied\nforce. To ensure accurate tactile feedback, a constant force must be applied to the object,\nwhich requires maintaining consistent internal actuation pressure. As shown in Fig. 2c\n(above) and 2d, once the gripper reaches the desired working pressure (14 PSI in this\nstudy), the tactile output remains stable and does not change over time after full force is\napplied.\nIn our system, the “threshold” is a user-defined value corresponding to the desired internal\ngripper pressure, while the “input pressure” refers to the actual internal air pressure\nmeasured by the onboard pressure sensor. The external air source is set to a much higher\nvalue than the desired pressure, and a closed-loop control system regulates the internal\npressure to maintain it at the target level.\nTo improve clarity, we have updated the flow chart by replacing “input pressure” with\n“internal gripper pressure” and “threshold” with “desired pressure.” Table 1 has also been\nrevised to explicitly illustrate the closed-loop control mechanism. We have updated the\nterminology for pressure and clarified the closed-loop control description in the text, and\nwe have revised Table 1 accordingly in the revised manuscript.\nIV. Fourth, did the authors explore usage of the tactile sensor data beyond optimising the\napplied air pressure to the gripper? (e.g. measure ripeness to decide whether to\nharvest yes/no?, or measure ripeness to decide whether to release in container A, B,\nor C?, or detect slipping during the rotational harvesting and increase the air pressure\napplied to the gripper if this occurs?)\nAuthor’s Response: We measured differences between ripe and unripe strawberries by\nrecording the tactile sensor response for strawberries at one day and three days post-\nharvest. The tactile sensors were able to clearly distinguish between the two, demonstrating\nthe potential to use this data for harvesting decisions. Combined grasping and harvesting\ndemonstrations are shown in Movies 12–13, where the system measures fruit ripeness in\nreal time.\nIn this paper, our primary focus was on semi-autonomous strategies, meaning that we\ndemonstrated real-time measurements and manually executed the picking step using the\nintegrated system. We did not conduct a fully autonomous harvesting test in this study;\nhowever, we have planned this for future work, where harvesting decisions will be made\nautomatically based on ripeness data. Our goal is to first establish the hardware and validate\nthese sensing capabilities and then extend them to autonomous control strategies in the\nfuture.\nFifth, for the ripeness detection, it feels as a missed opportunity to combine the ripeness\nestimation from the vision system with the ripeness estimation from the tactile sensors to get\na more accurate estimate. Again, it would be good to incorporate this in the flowchart.\nAuthor’s Response: This manuscript focuses on developing and integrating multimodal and\nmultifunctional capabilities into the gripper and on semi-autonomous validation. The use of\nstrawberries was intentional, as its ripeness is precisely estimated from color data. We performed\ntime-series tests on strawberries, quantifying color and corresponding fruit stiffness sequentially\n(Fig. 3a–c). From these data, we derived a correlation between vision and tactile measurements,\nallowing us to accurately predict ripeness from tactile measurements. We agree that vision and\ntouch combined is an opportunity to increase accuracy in predicting ripeness for fruit that is not\nclearly identified with color; however, we decided to apply tactile measurements for ripeness (in\na limited sense to a fruit that cannot be predicted by color (avocado). Now that our hardware is\nbuilt with initial demonstration, the significant effort of fusing the visual and tactile sensing for\naccurate ripeness estimation in fruit that can partially be estimated by color such as peaches and\nbananas.\n(c)\nFigure 3. Vision and tactile for firmness relationship\nSummarising, it would be good to have a flowchart for the harvesting task where it becomes clear\nhow all of the actuators and sensors are linked and together contribute to successful task execution.\nAuthor’s Response: Thank you for the detailed feedback. We have addressed all specific\nquestions, and Figure S22 now presents the updated flowchart depicting the complete process in\nthe revised manuscript.\n2. Several design decisions are not clear. Why did the authors opt for 5 fingers? Why the\ntapered shape? Why are the fingers aligned in the same plane? Please explain how\nthis choice of integration was found the most suitable for the harvesting task.\nAuthors Response: Thank you for raising this question. A stable grasp generally requires at least\nthree contact points. We therefore adopted a five-finger configuration to increase contact\nredundancy, distribute forces more evenly, and obtain more reliable tactile measurements given\nthe large variability in fruit size and shape. The additional fingers increase contact area and load\ncapacity; in our tests, the gripper lifted over 1 kg (~1,600% of its own weight). Keeping all fingers\nin the same plane enables interaction with a broad range of object sizes and shapes, as reflected in\nour workspace characterization, where the usable workspace expands by ~1,500% relative to its\nminimum configuration. Finally, the tapered/trapezoidal finger geometry is primarily aesthetic: it\nreduces inter-finger collisions during closing, allows full closure without gaps, and improves\ncontact coverage, while also providing a compact profile. Together, the multi-finger design and\nwide, well-characterized workspace support stable operation in dynamic conditions, which\nmotivated our design choice.\n3. Regarding the fibre sensors, Fig 5(a & c) show that the fibre makes a sharp bend\nbefore entering the light source and sensor. Could this explain the initial increase in\nlight intensity (i.e. that initial bending of the actuator allows the fibre to straighten a\nbit around those sharp bends)?\nAuthors Response: We appreciate the reviewer’s comment. Drawing on our lab’s longstanding\nexperience with stretchable optical-fiber sensors, we integrated multiple fibers into each finger\nwithout compromising bendability or baseline stiffness. Unlike prior grippers, where added\nsensors increased rigidity, our fibers are routed directly within the finger through guiding pathways\nthat preserve the finger’s compliance. The initial increase in light loss can be explained by our\nsensor assembly process. While integrating the fibers into the finger, there is micro-bending\nbetween the fibers and the clamps, due to minimal compressive loading. This slight bending\nproduces a characteristic response during finger bending at small bends, the stress relaxes, the\nbends straighten, and the optical signal changes gradually, whereas beyond a critical angle (~75–\n80°) the fibers enter tensile stretch, and the light intensity decreases sharply (see Fig. 5b–d). This\nbehavior was reproducible across seven independently fabricated fingers in our bending\ncharacterization. We have added an explanation in the revised manuscript.\n4. The sensor system may be prone to drifting. How constant were the sensors values\nover time (e.g. over 3 hours of continuous use or separate experiments one week\napart)? How did the authors address this issue\nAuthor’s Response: We did not observe notable drift in either the tactile or curvature sensors. We\ndid not observe any variance in continuous sensor testing as seen in the sensor characterization\n(Figures 5b, 6e, S13-16 in the manuscript). Each test was performed at least 7 times, with each test\ntaking a minute to run. The curves for each test overlap with each other and the standard deviation\nis very low. Additionally, the sensor’s baseline values did not significantly vary. For example, we\nused a 16-bit ADC, which has a range between 0 and 4.096V, to set the initial sensor output to 0.5\nV. From this value, we saw a very small change of 0.1V, which is 2.5% of the maximum sensor\noutput. We can mitigate this drift by resetting the baseline values to 0.5V using variable resistors\nin our experiments and by logarithmic normalization. We have reported this baseline drift in Figure\n4 by reporting the raw intensities and baseline values of two sensors on Day 1 and Day 3 of an\nexperiment.\nFigure 4. Sensor Drift for Tactile and Curvature Sensor\n5. In Figure 7 and the supplementary videos, the bellow segments appear to have a black\nmarking/cover that is not visible in figure 5 and 6. Does this serve a purpose?\nAuthor’s Response: The black covers were added to prevent leaks during the final videos\ndemonstrating the integrated system. They help minimize any damage from the sharp or dried\nstems of the plant while the gripper is harvesting fruits from it. Additionally, they reduce air leaks\nfrom high stress concentration areas (due fabrication inconsistency) of the gripper during long\noperational periods. During testing and evaluation, the risk of air leaks from environmental damage\nwas lower, so we did not add black covers to the gripper. Beyond preventing damage, they served\nno other purpose. We have added this clarification to the manuscript and relevant figure captions.\n6. The legend of Fig.6 (e) shows six different data series, but only three colours can be\nobserved in the plot. Do they fully overlap?\nAuthor’s Response: We apologize for this oversight. We mistakenly forgot to include the top\ndataset, but we have corrected this in the revised version.\n7. Please check the paper for some small typos (e.g. Table 1--> \"volatge, \"clossed\",\n\"valtage\", \"distnace\", line 233 \"both had hues are\")\nAuthor’s Response: Thank you. We carefully rechecked the manuscript and have, hopefully,\ncorrected all typographical errors in the revised version.\n8. Have figure S6 row 1, middle and right image been accidentally swapped?\nAuthor’s Response: Thank you for pointing this out. We have fixed it in the updated version.\nOverall, this research work has potential for being published in Nature Communications, but the\nmanuscript feels incomplete at this stage. A stronger link between the sensors and actuators would\ngreatly enhance the impact and contribution of the manuscript.\nAuthor’s Response: Thank you again for the thorough feedback. We believe the revised\nmanuscript addresses all points raised.\nReviewer #2 (Remarks to the Author):\nThe reviewed work addresses an important challenge in robotics -- soft visuo-tactile manipulation.\nPhysical intelligence through soft robotics that you are tackling is right on time.\nThe comments are attached.\n1. Research Area and Journal\nThe paper tackles a narrow, yet very important research area in Robotics –tactile sensing. There\nare similar review papers in this journal, for example [Sanchez J, Corrales J-A, Bouzgarrou B-C,\nMezouar Y. Robotic manipulation and sensing of deformable objects in domestic and industrial\napplications: a survey. The International Journal of Robotics Research. 2018]. In this connection,\nI believe that this new review paper may find interested readers of this Robotics journal.\nAuthor’s Response: We thank the reviewer for bringing this article to our attention. We have\nconducted a comprehensive review of related gripper technologies and prepared a comparative\nplot illustrating the current state of the art in soft robotic grippers with multiple sensing modalities\n(see Figure 1). We have also cited the recommended article for the reader’s reference and included\nthe citation and figure 1 in the revised manuscript.\nFigure 1. Comparative plot shows different finger designs, sensing modalities, and number\nof sensors embodied.\n2. Manuscript\nLine 209 Gripper characteristics. The authors performed experiments on grasping forces in\nresponse to input pressures and the size of objects. It would be desirable to measure the\npulling force of a single finger in response to its configuration. For example, in the finger of\nan anthropomorphic robot hand, its joint angles determine the maximum pulling force [1].\nThe following literature is random; it is given as an example only: [1]\nDOI:10.1109/ICMA.2013.6618171\nAuthor’s Response: Thank you for raising this point. In Fig. 3c, we report the pulling (pull-off)\nforce—that is, the force required to remove an object from the gripper’s grasp—measured over\ntime and across different object sizes. This measurement is also shown in Movie 8 and\nSupplementary Fig. S3. In the original draft, this metric was inadvertently labeled as “grasping\nforce.” We have now corrected the terminology throughout the manuscript, figure, and caption to\n“pulling (pull-off) force.” In addition, we evaluated vertical loading by progressively adding\nweight until the gripper could no longer sustain the load. Thus, horizontal and vertical loading\nconditions were tested for active pulling forces and gravitational pull respectively. Finally, we also\nmeasured the blocking force for individual fingers, defined as the compressive force exerted by a\nfingertip against a rigid surface at a given internal pressure. Together, the pulling and blocking\nforce measurements provide complementary tensile (pulling) and compressive (blocking)\ncharacterizations of the gripper. Relevant details and results have been included in the revised\nmanuscript and Supplementary Materials.\nLine 225: What were the ambient light conditions? A reference is needed; it could be\nprovided by a luxmeter. Which hyper-spectral camera was used to state σ.\nAuthor’s Response: Thank you for this comment. We now report the ambient illuminance (lux)\nat work bench and a couple of feet above, measured with a handheld luxmeter (Dr. meter,\nLX1330B). Measurements were taken at the gripper test bench with the sensor pointed toward the\nlight source. The recorded values were ~421 lux at the bench and ~832 lux a couple feet above the\nbench. These values have been added to the Methods and to the caption of Fig. 4.\nWe did not use a hyperspectral camera. The symbol σ refers to the standard deviation of hue\ncomputed from RGB images. Specifically, we used a Raspberry Pi RGB camera module, captured\nimages with fixed light exposure, converted RGB to HSV, segmented the fruit region of interest,\nconstructed the hue histogram (PMF), and computed the mean and standard deviation of hue. We\nhave clarified this in the methods, and we have stated the exact camera, optics, and capture settings\nin methods and supplementary information.\nLine 243: 100 per cent of the success rate is stated. How many trials were made? In overall,\nstatistical data is missing. It is desirable to see plots that illustrate the average and standard\ndeviation of several trials.\nAuthor’s Response: For each shape, we conducted at least five trials and recorded a binary\noutcome (success/failure). Because success rate is a binomial proportion, we report the number of\nsuccesses and trials together with 95% Wilson confidence intervals rather than a standard\ndeviation. For n=5, a 100% success (5/5) yields a 95% Wilson CI of [0.57,1.00], whereas an 80%\nsuccess (4/5) yields [0.38,0.96]. Pooled across all instances in this experiment (59 successes in 60\ntrials), the overall success rate is 0.983 with a 95% Wilson CI of [0.91,1.00]. We have added these\nintervals to the revised manuscript and figure captions.\nLine 251 Proprioceptive sensing system, which is named as” Curvature sensor” by the\nauthors, should follow existing terminology.\nAuthor’s Response: Thank you for the suggestion. We have defined the curvature sensor as a type\nof proprioceptive sensing in the manuscript.\nLine 265 As a reader who knows a little bit about optical fibers, I started to doubt the\nexperimental results presented in the paper. There are coated optofibers, there is a stray\nlight; and the light intensity variation must be revised. The increase and drop of the light\nintensity could be due to misalignment at the source of the light and the photoresistor.\nAuthor Response: Thank you for this comment. We incorporate several design features to mitigate\nambient-light effects and prevent source–detector misalignment. The optical fibers are bonded into\nrigid holders, and the holders are bonded to the fingers, so the relative geometry is fixed during\noperation. The light source and photodetector (LED and phototransistor) are rigidly mounted to\nthe holder and encapsulated with an opaque silicone sealant, which blocks stray light and further\nstiffens the source–fiber and fiber–detector interfaces against vibration or shift.\nWe use IR LEDs with matched detectors, and the fibers guide this light by total internal reflection\n(but with a critical angle lower than in plastic or glass due to fewer material options). Because the\nemitting/detecting elements are enclosed and spectrally separated from typical ambient\nillumination, the sensor output is largely insensitive to room lighting. This is illustrated in Fig. 2,\nwhere signals recorded under bright and dark conditions are nearly identical.\nRegarding the initial increase and drop of light intensity, this is due to how the optical fibers were\nassembled. During assembly, a small amount of IPA was used to seat the fibers in the tight-fitting\nchannels; after bonding the leads to the LED and phototransistor, the fibers remained under slight\npretension. This produces a characteristic response: at small bends, the pretension relaxes and the\noptical signal increases gradually, whereas beyond a critical angle (~75–80°) the fibers enter tensile\nstretch, and the light-intensity change increases sharply (see Fig. 5b–d). This behavior was\nreproducible across seven independently fabricated fingers in our bending characterization, so we\nare confident it is not the result of stray light entering the fibers.\nFor completeness, we have added the measured ambient illuminance (lux) at the testbench to the\nmethods and relevant figure captions, and we clarify the mounting/encapsulation details in the\nrevised manuscript.\nFigure 2. Effects of external light conditioning on sensor performance.\nThe further review of the manuscript will be meaningful after the elaboration of the\nproposed curvature sensing system. Especially, line 233, the text and the statements must\nbe carefully revised:” (strawberry-1 and -2) both had hues are 30.01 ...”. Moreover, the\nrange of the Hue value is too large.\nAuthor Response: In OpenCV, the scale for Hue measurements is 0 – 179, while the theoretical\nvalue for red varies between 0 – 10 and 170 – 179. In our case, the strawberry is not a standard red\ncolor as the strawberry has green leaves, yellow seeds, and different colorations across the fruit.\nFrom Figure S6, the measured distribution is wider because the fruit is not a single color, and the\nvalues may wrap around from 170 to 50. Furthermore, as seen in Figure S6, two strawberries with\ndifferent colors can have similar color distributions. This issue causes a wide range of hue values.\nWhen we recorded the same distribution with standard colors, the distribution for red was very\nsharp at 172 with a smaller deviation of 25. The green had a sharp peak at 70. Since the strawberries\nhad a mix of red and green coloring, a hue value of about 30 with a wide distribution makes sense\nand is consistent with the base hue measurements of red and green.\nIn Line 236, which value did author mean: ”the saturation and value...” ? The intensity\nwould depend a lot on the ambient light.\nAuthor Response: Thank you for the response. We have clarified this sentence in the revised\nmanuscript and added a further explanation below.\nThe saturation and value measurements were inconclusive because of the constant lighting\nconditions that were used. In color measurements under constant illumination, the hue component\ngenerally exhibits greater variability than saturation or value due to its mathematical dependence\non the relative ratios of the RGB channels. Different colors will cause very different hue values.\nIn contrast, saturation is defined by the relative difference between the maximum and minimum\nchannel intensities and value is defined as the maximum channel intensity. Under constant lighting,\nthese two metrics will remain stable, making them less useful for differentiating object colors\ncompared to the hue channel.\nWe would like to thank reviewer 2 for the comprehensive review and feedback for our paper. We\nappreciate the reviewer’s time for taking their time to improve our manuscript.\nReviewer #3 (Remarks to the Author):\nThe authors present a soft pneumatic gripper with integrated proprioceptive and tactile\nsensing. By combining these capabilities with vision, the gripper can assess fruit ripeness and\nperform grasping/picking tasks without causing damage. Thorough characterization of the\ngripper has been conducted and its potential application in fruit picking has been\ndemonstrated.\nOverall, the paper is well-written and organized. However, the following issues need to be\naddressed before it can be considered for publication in Nature Communications:\nAuthors Response: We thank the reviewer for the thorough feedback and appreciate the\nopportunity to address the points raised. Here are our responses to the specific questions raised by\nthe reviewer.\n1. The comparison with related work is insufficient. The authors claim that the proposed\nmethod (collocated camera and ToF sensor + tactile sensing + proprioception) is better\nthan traditional methods (camera + tactile sensing), e.g., for handling occlusions, but\nthe strawberry picking demonstration does not substantiate this advantage over\nexisting techniques. A more detailed comparative analysis, backed by stronger\nexperimental evidence, is required to highlight the method’s novelty and justify its\nclaims.\nAuthor Response: We appreciate the reviewer’s feedback. To contextualize our\ncontributions, we conducted a detailed literature review and added a comparison plot (Fig.\n1) that contrasts prior soft grippers and our system in terms of the number of embedded\nsensors, the sensing modalities integrated, and the resulting capabilities.\nRegarding the benefits of our approach relative to existing methods: many current systems\npair a monocular camera with a single tactile sensor. A monocular camera provides color\nand shape but not absolute size without additional hardware or assumptions; obtaining\nreliable size typically requires a depth camera or LiDAR, which are bulkier and most often\nmounted on mobile platforms rather than collocated on the gripper. In contrast, our vision\nmodule matches the gripper’s form factor and is collocated at the end effector, enabling\nconcurrent measurement of color, shape, and size. Quantifying contact when both bodies\nare soft or compliant is also challenging. We address this by distributing multiple tactile\nsensors across the fingers and coupling their outputs with proprioceptive curvature\nsensing, so the controller knows each finger’s bending state together with the\ncorresponding tactile response (Fig. 6 and Supplementary Data). Taken together, these\nchoices provide the set of modalities needed to capture object features as they exist in\nbiological systems, which have not yet been fully integrated into soft grippers at this scale.\nConcerning occlusion, we performed both benchtop and in situ experiments with the\ncollocated vision module while the fully functional gripper operated. We characterized\ncolor, size, and shape and observed a shape classification accuracy of 100%, a size\nestimation error below 1.8%, and correct identification of six distinct colors, beside\nwithout any intelligent vision algorithm (Fig. 1). Prior work on collocated vision in rigid\ngrippers has reported improved occlusion handling due to higher visual dexterity, which\naligns with our design rationale and observations [1]\nReal-time tests during grasping and harvesting yielded consistent results. While this paper\nconcentrates on developing a fully sensing soft gripper and demonstrating semi-\nautonomous, human-triggered picking, our next step is a fully autonomous control\nstrategy that fuses vision- and tactile-derived ripeness estimates.\nFigure 1. Comparative plot shows different finger designs, sensing modalities, and number\nof sensors embodied.\n2. The light intensity loss curve in Fig. 5d is non-monotonous, which creates ambiguity, as a\nsingle intensity value could correspond to two different bending angles. The authors should\ndiscuss how they resolve this ambiguity in practice. Furthermore, during grasping, the\ncurvature sensor may come into contact with the fruit. It is unclear how this unmodeled\nphysical contact would affect the sensor’s accuracy, and a discussion on the curvature\nsensor’s robustness to such interactions is needed.\nAuthor’s Response: We understand the reviewer’s concern. The raw optical loss–versus–\nangle data exhibit a small non-monotonic loop caused by the transition from pre-tension\nrelease to tensile stretch (see Fig. 5d). Because bending angle is coupled to internal\npressure, the optical signal includes pressure-dependent gain. Normalizing by internal\npressure (or, equivalently, subtracting the per-pressure baseline) removes this gain and\nyields a monotonic mapping with angle (Fig. 2). For calibration and control, we therefore\n(i) operate at fixed internal pressure or (ii) use the normalized signal, which is monotonic\nin angle.\nFigure 2. Bending angle with normalized light loss\n3. Given the fingertip sensor’s relatively large contact area, different regions of the sensor\nwill contact the fruit during picking. How would the tactile sensor’s output (as shown in\nFig. 6f) be affected by variations in the object’s shape, size, and the specific location of\ncontact? And how would the sensor respond to unexpected shear forces, such as those from\nexternal disturbances during or after a picking motion?\nAuthor Response: Thank you for this question. The fingertip contact areas are\napproximately 1.68 cm² (top) and 1.55 cm² (side), while the average surface area of a\nstrawberry is about 14 cm². Thus, each fingertip covers only a relatively small portion of\nthe fruit surface, and variation across fingers is minimal. The fingertip sensor measures\noptical loss generated by local normal compression of the soft pad and embedded fiber;\nthe readout therefore represents a spatially averaged normal load over the contact patch.\nObject shape/size and contact location can change the patch geometry and moment arm.\nWe mitigate this in two ways: (i) we regulate the internal pressure to a fixed setpoint to\nstabilize normal force, and (ii) we pair the tactile signal with proprioceptive curvature,\nso the finger’s bending state is known. In practice, we begin each trial with a small\napproach gap and align the fingertip to contact approximately normal to the surface; under\nthese conditions, conditioning the tactile signal on the measured curvature yields\nconsistent responses across different contact regions and fruit radii (see Fig. 6f and\nSupplementary Fig. S24-26).\nWith respect to shear or external disturbances, the present sensor is most sensitive to\nnormal load, but shear produces a characteristic transient: a rapid change in optical loss\naccompanied by a curvature change as the contact centroid shifts. During semi-autonomous\ntests, the measurement phase was performed with normal approach, and we did not observe\nslip; during picking, once all five fingertips established normal contact, fruit removal\nproceeded without slipping. In future work, we will use simple rate- and slope-based\nfeatures to flag these shear/disturbance events and trigger either a brief pressure increases\nto re-establish contact or a commanded re-grasp.\n4. The optical fiber for tactile sensing is continuous across the top and side pads, yet these\nregions are characterized independently (Fig. 6f). Please clarify how the sensor responds\nwhen forces are applied to both the top and side simultaneously.\nAuthor Response: We have not experimentally tested simultaneous top-and-side loading;\nall tests in Fig. 6f were performed with normal contact on a single face (top or side) at a\nfixed internal pressure. The fingertip uses a single continuous U-shaped fiber spanning\nboth pads, and the output is the total optical loss integrated along this path. Based on the\nmicro-bending/localized-deformation mechanism, losses from multiple contact regions are\nexpected to add approximately linearly, so combined loading should be scaled with the\namount of fiber that is deformed rather than the exact contact location. In our control policy,\nthe gripper is ensured to establish normal contact on either pad face, making simultaneous\nloading of both faces unlikely during the measurement phase.\n5. Fig. 6d reveals a noticeable hysteresis in the sensor’s response during loading and\nunloading cycles. This could significantly affect the accuracy of the proposed model.\nPlease provide a more detailed analysis of this behavior and its implications.\nAuthor Response: Thank you for raising this point. In Fig. 6d, we report the force–\ndisplacement response for Sensor 1 under indentation with a rigid probe; as expected for a\nsoft viscoelastic pad, this mechanical curve exhibits a hysteresis loop. However, when we\nplot the optical readout against either applied force or displacement, the curves for both the\ntop and side contacts are essentially single-valued with no observable hysteresis (Fig.\nS12c–d). The same behavior holds for Sensors S2–S5 (Fig. S13c–d, g–h; S14c–d, g–h;\nS15c–d, g–h). These results indicate that hysteresis in the mechanical substrate does not\nmeasurably degrade the accuracy of the optical tactile readout in our operating regime.\nBecause the present paper focuses on system integration and semi-autonomous\ndemonstrations, a detailed constitutive model of viscoelastic and rate-dependent effects is\nbeyond scope of this work.\n6. What’s the purpose of measuring perimeters of objects (e.g., in Fig.. 4d)? Just for\nquantifying the accuracy of the object size estimation method, or does it contribute to the\nsubsequent grasping/picking strategy?\nAuthor’s Response: Thank you for the question. In Fig. 4d we report object perimeter as\na 2D size metric derived from the edge map. Its purpose in this paper is to quantify the\naccuracy of our vision-based size estimation: it provides a consistent scalar for statistical\nanalysis and error metrics across objects of different shapes and colors. At present, these\nperimeter measurements do not drive the grasping/picking strategy; the demonstrations are\nsemi-autonomous with a fixed control policy and manual trigger. In future fully\nautonomous operation, the size estimate (perimeter and derived diameter/area) will be\nfused with tactile/proprioceptive data to inform grasp aperture and approach pose, set\npressure setpoints, and enable size-based sorting.\n7. Please provide clear definitions for the “workspace area” in Fig. 2e and the “blocking\nforce” in Fig. 3a.\nAuthor’s Response: For clarity, we define our terms as follows. Workspace area is the\nplanar reach envelope of the gripper. For each internal pressure, we recorded the distal\nfingertip positions and approximated the boundary by a circle passing through the\noutermost tip locations; the area is then A=πr2. The maximum workspace corresponds to\nthe flat state and the minimum to the fully folded state. Blocking force is the normal force\nthe fingertip exerts on a rigid surface at a given internal pressure; we measured it by\npressing the fingertip against a calibrated scale and recording the steady-state reading.\nThese definitions have been added to the revised manuscript for clarity.\n8. The manuscript frequently refers to object sizes, such as “a 2 cm object”, without\nspecifying whether this refers to diameter, side length, or another dimension. This\nambiguity should be resolved. Please define these terms clearly and consider annotating\ndimensions directly on the objects in Figs 3d or 3e. Please check and fix this issue\nthroughout the manuscript.\nAuthor’s Response: We have added explicit definitions when referring to the size of the\nobjects in the manuscript, supplementary information, and figure captions. In this figure,\nthis 2 cm refers to the diameter of the hemispherical object.\n9. Fig. 3d is confusing to read at first glance. I suggest using different colors to indicate the\nreal object from the error bars.\nAuthor’s Response: Thank you for your question. We have modified this in the revised\nmanuscript.\n10. The meaning of “80mm, 100mm, and 120mm” in Figs. 4d and 4e is unclear. Please add\nexplicit annotations, for example, using “d=80mm” to denote diameter.\nAuthor’s Response: We have added explicit definitions to the revised manuscript. In this\nfigure, 80mm, 100mm, and 120mm refers to the distance between the camera and the\nrespective objects. We have updated the manuscript and figure captions to reflect this.\nReviewer #3 (Remarks on code availability):\nThe provided GitHub page does not exist.\nAuthor’s Response: We have changed the GitHub setting. It will be available to public access\nnow. The code is available at this link: https://github.com/mishra-anand/Gripper_code.\n[1] Chaudhury, A. N., Man, T., Yuan, W. & Atkeson, C. G. Using collocated vision and tactile\nsensors for visual servoing and localization. IEEE Robotics and Automation Letters 7, 3427-3434\n(2022)."}, {"round": 2, "reviewer_comments": {"reviewer_1": "The authors have appropriately addressed all of the reviewer's previous concerns. The reviewer enjoyed reading this interesting manuscript while also looking forward to the proposed follow-up research. The reviewer recommends the manuscript for publication in Nature Communications in its current form.", "reviewer_2": "1 Revised manuscript\nIt took me a while to start working on the review to provide the feedback. I\nread the revised manuscript and the reply letter much later than it was received.\nThe revised manuscript was improved a lot by the authors. They replied\nto my comments well. I still had doubts about the optofibers and the bending\nangle. Nevertheless, in this paper we are not expecting the precision that is\nneeded for medical robots.\nThe authors replied to other reviewers and did new experiments.\nIn the future, I hope to see the gripper and the sensing system integrated\nusing flexible PCBs to get rid of the canopy of electric wires, which may become\na bottleneck for dynamic object manipulation.\nIt is well done.\nIt is acceptable for the proposed application.\nIt includes the information that was missing before.", "reviewer_3": "I thank the authors for their work on the revision. While I appreciate the changes made, there are still some issues from the previous review that are not fully addressed.\n1. For the first point, I acknowledge the addition of the new comparison in Figure S28. However, a comparison based on the number of sensors and modalities is not sufficient to highlight the contribution of this work. Instead, the authors should compare the performance of their proposed sensors against existing systems to provide a more convincing analysis.\nRegarding occlusion, the authors claim in their response that the system minimizes this issue. The revised manuscript, however, lacks the explicit experimental results necessary to substantiate this claim. To strengthen the paper, evidence demonstrating the system's performance under occlusion should be included.\n2. In the author’s response to the second point, the new figure showing the bending angle with normalized light loss is an excellent addition. It successfully answers my question and clarifies the ambiguity of the original Figure 5d. This clarification is very important for all readers, not just the reviewer, and therefore it should be integrated into the revised manuscript.\nIn addition, there appears to be no response to my second question from this part: “Furthermore, during grasping, the curvature sensor may come into contact with the fruit. It is unclear how this unmodeled physical contact would affect the sensor’s accuracy, and a discussion on the curvature sensor’s robustness to such interactions is needed.” This point remains unaddressed and requires a discussion in the text."}, "author_response": "Authors’ Response: We thank the reviewer for the thoughtful and in-depth feedback, which has\nsubstantially helped improve our manuscript. We are pleased that the reviewer is satisfied with the\nchanges made in the revision.\nAuthors’ Response: We are genuinely grateful for the reviewers’ feedback and are pleased that\nthe reviewer is satisfied with our revisions. We also plan to incorporate flexible and reduced wiring\nin future versions of the gripper.\nAuthors’ Response: We thank the reviewer for the thorough feedback. We are pleased that we\nwere able to successfully address most of the reviewer’s concerns and appreciate the opportunity\nto respond to the remaining points. Below, we provide detailed responses to each specific question\nraised.\n1. For the first point, I acknowledge the addition of the new comparison in Figure S28.\nHowever, a comparison based on the number of sensors and modalities is not sufficient to\nhighlight the contribution of this work. Instead, the authors should compare the\nperformance of their proposed sensors against existing systems to provide a more\nconvincing analysis.\nAuthors Response: We appreciate the reviewer’s acknowledgement of Figure S28. In\npreparing this comparison figure, we initially attempted to construct a figure that directly\nevaluated our sensor’s performance against existing systems. During this process, we found\nthat most publications describing sensorized grippers evaluated their performance with\ngripper specific metrics. In many cases, analyses were limited to image classifications or\nreporting voltage, pressure, or force measurements specific to individual sensor designs,\nmaking it hard to compare sensor performance across different sensor designs.\nDespite an extensive review of the literature, we were unable to identify studies that\nreported standardized metrics relevant to our multimodal system, such as color accuracy,\nsize accuracy, shape reconstruction accuracy, curvature-sensing accuracy, pressure\nmeasurement accuracy, or tactile feedback performance. As these metrics are not evaluated\nuniformly across prior work, it was not feasible to construct a meaningful performance\nchart that compared our system directly with others. Furthermore, many of the sensors used\nin these grippers are custom-built and fundamentally different in their operating principles,\nmaking cross-system evaluation inherently difficult.\nIn addition, we could not find any prior work that incorporates stretchable light-guide-\nbased optical fibers within a gripper to enable multimodal sensory outputs, except one of\nour previous works. This absence of directly comparable systems prevents a one-to-one\nperformance comparison but also underscores the novelty of our approach.\nGiven these constraints, we presented a comparison based on sensing modality and sensor\ncount, as these parameters were unambiguously reported across existing studies. We also\nrealized that most of the existing grippers did not have as many sensing modalities as our\ngrippers, so we wanted to highlight this in a comparison figure. Looking forward, our aim\nis to establish a clearer framework for benchmarking multimodal soft grippers. By\nexplicitly reporting key modalities—including shape, size, color, touch, and\nproprioception, we hope this work will help shape future standards for evaluating grippers\nboth in laboratory environments and in field applications.\n2. Regarding occlusion, the authors claim in their response that the system minimizes this\nissue. The revised manuscript, however, lacks the explicit experimental results necessary\nto substantiate this claim. To strengthen the paper, evidence demonstrating the system's\nperformance under occlusion should be included.\nAuthors Response: We thank the reviewer for the thoughtful feedback on our response. In\nthis paper, our primary objective was to demonstrate a soft gripper with integrated multi-\nmodal sensing capabilities. Achieving simultaneous measurement of shape, size, color,\ntactile cues, proprioception, pressure, and distance within a single compact soft gripper has\nhistorically been extremely challenging. We did not focus this work on occlusion, instead\nwe observed experimental evidence, such as high shape and size accuracy, that suggested\nour system improved occlusion. One of our main contributions is to show that a collocated\ncamera can be embedded within the gripper and fused with curvature, pressure, and\nproprioceptive signals to generate a unified multimodal sensing output.\nImportantly, our aim was not to verify that the collocated camera explicitly minimizes\nocclusion in clustered or outdoor environment. Instead, we used the collocated camera to\nextract vision data and demonstrated that the system achieves very high performance (100\npercent for shape, size, color, and distance estimation on standard known objects). Because\nthis performance can only be achieved if the objects are not obstructed, this performance\nindirectly suggests that the collocated configuration reduces self-occlusion. This\nobservation is consistent with prior work in a rigid gripper using a collocated camera,\nwhich similarly reported reduced occlusion [1]. At the time, we did not anticipate the need\nfor dedicated occlusion experiments to quantify this effect in our system.\nWe agree that explicit occlusion testing will be essential in agricultural environments,\nwhere occlusion is common. This naturally presents a direction for future work involving\nfield tests under controlled and natural occlusion conditions. However, such work is\nbeyond the primary scope of the present study, which focuses on the design of multimodal\nsoft sensing platform and demonstration on plants with real fruits and standard objects\n(cube, sphere, ....) in a laboratory setting.\n3. In the author’s response to the second point, the new figure showing the bending angle with\nnormalized light loss is an excellent addition. It successfully answers my question and\nclarifies the ambiguity of the original Figure 5d. This clarification is very important for all\nreaders, not just the reviewer, and therefore it should be integrated into the revised\nmanuscript.\nAuthors Response: We thank the reviewer for the thorough feedback and have integrated\nthe figure and explanation into the revised manuscript. The new figure is Figure S12, and\nthe text has been inserted into the revised manuscript.\n4. In addition, there appears to be no response to my second question from this part:\n“Furthermore, during grasping, the curvature sensor may come into contact with the fruit.\nIt is unclear how this unmodeled physical contact would affect the sensor’s accuracy, and\na discussion on the curvature sensor’s robustness to such interactions is needed.” This point\nremains unaddressed and requires discussion in the text.\nAuthors Response: We sincerely apologize for this oversight. We unintentionally missed\nresponding to this question in our previous revision. There are two regions where the\noptical fiber could, in principle, come into contact with an external object.\nThe first region is the palmar side of the finger, where the optical fiber is routed. To mitigate\ndirect contact in this area, the gripper includes soft finger clamps distributed along the fiber\npath. These clamps provide a compliant protective layer above the curvature sensor,\nsubstantially reducing the likelihood of direct contact between the optical fiber and grasped\nobjects. The second region is the interface between the optical fibers and the\nLED/phototransistor holders. By design, we positioned the collocated camera and distance\nsensor at the same location where the five LED/phototransistor holders converge. This\nmounting structure sits above the holders and thus acts as an elevated contact point,\nensuring that objects touch this protective mount rather than the optical fibers themselves.\nAdditionally, the accuracy of the curvature sensor could be affected by sustained contact\nwith an external object or force. However, because the gripper incorporates five\nindependent curvature sensors, cross-sensor validation can be implemented to identify\nanomalous signals. If one sensor is perturbed due to incidental contact, its reading can be\ndiscarded or corrected using the outputs from the remaining sensors. Since all sensor data\nare already accessible, incorporating outlier-rejection algorithms represents a logical next\nstep to further improve the robustness of the curvature-sensing system.\nAcross all experiments involving real fruits, fruits attached to plants, and standard objects,\nwe did not observe any instances in which objects contacted the curvature-sensing fibers.\nNevertheless, we also believe that under certain conditions—such as interactions with\nsmall objects or semi-solid or fluidic materials—contact could occur and potentially\ninfluence curvature estimation. During our experiments, the optical fibers were routinely\ncleaned to remove particulates or liquids that may have accumulated during prior tests or\nenvironmental exposure.\nIn future designs, we plan to address this limitation by integrating protective cladding and\nmore robust mechanical enclosures around the sensing regions, implementing positive-\ncontrol evaluations for the curvature sensors, and incorporating outlier-rejection algorithms\ninto the sensing pipeline. We have now included a discussion of this limitation and the\ncorresponding mitigation strategies in the Conclusion section of the revised manuscript.\nAuthor’s Response: We have changed the GitHub setting. It will be available to public access\nnow. The code is available at this link: https://github.com/mishra-anand/Gripper_code.\n[1] Chaudhury, A. N., Man, T., Yuan, W. & Atkeson, C. G. Using collocated vision and tactile\nsensors for visual servoing and localization. IEEE Robotics and Automation Letters 7, 3427-3434\n(2022)."}, {"round": 3, "reviewer_comments": {"reviewer_3": "The authors have addressed all of my concerns. I don't have further comments."}, "author_response": ""}]

Decision letter