Visual Learning Towards Soft Robot Force Control using a 3D Metamaterial with Differential Stiffness
Authors: Fang Wan
Xiaobo Liu
Ning Guo
Xudong Han
Feng Tian
Chaoyang Song
Venue: CoRL2021 Poster (2021)
Keywords: Visual Learning
Soft Robot
Force Control

Abstract: This paper explores the feasibility of learning robot force control and interaction using soft metamaterial and machine vision. We start by investigating the differential stiffness of a hollow, cone-shaped, 3D metamaterial made from soft rubber, achieving a large stiffness ratio between the axial and radial directions that leads to an adaptive form response in omni-directions during physical interaction. Then, using image data collected from its internal deformation during various interactions, we explored two similar designs but different learning strategies to estimate force control and interactions on the end-effector of a UR10 e-series robot arm. One is to directly learn the force and torque response from raw images of the metamaterial's internal deformation. The other is to indirectly estimate the 6D force and torque using a neural network by visually tracking the 6D pose of a marker fixed inside the 3D metamaterial. Finally, we integrated the two proposed systems and achieved similar force feedback and control interactions in simple tasks such as circle following and text writing. Our results show that the learning method holds the potential to support the concept of soft robot force control, providing an intuitive interface at a low cost for robotic systems, generating comparable and capable performances against classical force and torque sensors.

Visual Learning Towards Soft Robot Force Control using a 3D Metamaterial with Differential Stiffness
Fang Wan AncoraSpring, Inc.
1001 Xueyuan Avenue, Shenzhen, China 518055
sophie.fwan www.wettawwaa........omoommaa... com
Xiaobo Liu, Ning Guo, Xudong Han, Feng Tian, Chaoyang Song*
Southern University of Science and Technology 1088 Xueyuan Avenue, Shenzhen, China 518055 (11930807, 11930729, 11812519, 11910826] www.hl/ssssss.h....ccc wettechh.eu..com cn, songcydieee.org
Abstract: This paper explores the feasibility of learning robot force control and interaction using soft metamaterial and machine vision. We start by investigating the differential stiffiess a hollow, cone-shaped, 3D metamaterial made from soft rubber, achieving large stiffness ratio between the axial and radial direc- tions that leads to an adaptive form response in omni-directions during physical interaction. Then, using image data collected from its internal deformation dur- ing various interactions, we explored two similar designs but different learning strategies to estimate force control and interactions the end-effector of UR10 e-series robot arm. One is to directly learn the force and torque response from raw images of the metamaterial's internal deformation. The other is to indirectly esti- mate the 6D force and torque using a neural network by visually tracking the 6D pose of a marker fixed inside the 3D metamaterial. Finally, we integrated the two proposed systems and achieved similar force feedback and control interactions in simple tasks such as circle following and text writing. Our results show that the learning method holds the potential to support the concept of soft robot force con- trol, providing an intuitive interface low cost for robotic systems, generating comparable and capable performances against classical force and torque sensors.
Keywords: Visual Learning, Soft Robot, Force Control
Introduction
Force feedback and force control become mandatory to achieve a robust and versatile interaction between robotic system and the physical enviromment, well as safe and dependable operations in the presence of human. Direct enhancement in sensory feedback from force and torque sensors provides robotic systems with the capability to model and react to the various physical contacts [1, 2, 3]. Traditional sensing methods for force and torque rely heavily on rigid-body theories to measure and directly react to the contact dynamics. This paper aims to explore the concept of soft robot force control by using learning-bast
methods with visual information collected from a soft, 3D metamaterial as robotic interface for human-robot interactions.
Sensory integration for rigidly-built robots is usually convenient design and model using classical theories in robotics, which usually output interaction measurement of a specific point in time-series [4]. On the other hand, soft robots made from compliant materials usually show life-like, adaptive motions at a relatively low cost. However, they are challenging to model due to their non-linear mechanics, making difficult integrate advanced sensory. Instead, field sensing solutions, such as light [5], magnetism [6], and vision [7, 8), are usually adopted for soft robots for proprioceptive sensing [9], which promotes the application of learning- based solutions to overcome the modeling challenge [10].
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

3D metamaterials are artificial composite structures in three dimensions with exotic material prop- erties [11], aiming effective material parameters that go beyond those of the ingredient ones, capable of generating desirable response through engineering design [12]. For example, recent re- search shows an optical lace design [5], where optical fiber networks are integrated with various metamaterial structures to achieve effective sensory integration for human-robot interaction. On the other hand, recent development in meta-neural neewwrk provides a novel solution for passive object recognition by acoustics using metamaterial's 's unit-cells to produce sueesuubwaell phase shift as training parameters [13]. Periodic properties, such as electromagnetic, optical, acoustic, and me- chanical ones, provide 3D metamaterials with the engineering potential towards distributed sensing for advanced robot control, where learning-basee based methods become an effective tool for end-to-end integration [14].
The contributions of this paper are listed as the following.
The investigation of soft, 3D metamaterial with differential stiffness for passive form adaptation in omni-directions in the radial plane:
A direct rision-based sensory integration of the 3D metamaterial for estimating the force and torque using a modified resnet;
The design integration of a marker inside the omni-adaptive metamaterial to facilitate effi cient, accurate, and real-time 6D force and torque using vision-based pose tracking;
System integration towards soft robot force control with comparable performance against commercial force and torque sensors with much-reduced hardware cost and complexity.
This paper is organized follows. In section 2, we introduce class of 3D metamaterial with omni- directional form adaptation, investigate its differential stiffness properties, and propose a sensory integration by adding camera at the base. In section 3, we use different methods to learn and estimate the rigid-soft interactions of the 3D metamaterial and experiment with a URI0 e-series robot for some basic force control tasks. Conclusion, limitations, and future works are enclosed in the final section, which ends this paper.
2 A Soft, 3D Metamaterial with Differential Stiffness
2.1 3D Metamaterial Design with a Soft Network Structure
We adopt network structure made from soft rubber in a hollow, cone-shaped 3D metamaterial as the design of interest, which has been researched in few recent publications to demonstrate its omni-direction adaptation for grasping [15] and sensing [16, 17]. As shown in Fig. 1, the proposed 3D metamaterial features gradually shrinking cross-section from the base to the tip, which can be designed with various geometric layouts based on the desired adaptation and form factor. Each layer comprises several nodes that are linked by soft, compliant rubber bars. The gradually shrinking design usually ends at the tip, which can be single multiple nodes.
As illustrated in Fig. the proposed 3D metamaterial capable of three major adaptive modes, including surface adaptation, twisted adaptation, and edge adaptation in all directions in the radial plane. Due to material softness, the 3D metamaterial also exhibits smooth transition between the three adaptive modes during physical contact. The adaptive effects, which apply to all three adaptive modes, are observed to be the most significant around the central area along the axial direction, but less so around the base with sizeable passive reaction force and less so around the tip with less passive reaction force. During the process of form adaptation, due to the shrinking design characteristics, the tip of 3D metamaterial generally remains (or near) the exact location with only spatial twisting, exhibit a passive form-closure effect that is similar to caging in grasping, making simple, convenient, and practical design for robotic manipulation with unstructured environment [15, 17]. On the other hand, the network structure with a hollow space inside provides abundant design space for sensory integration [16, 18).
2.2 Radial Stiffness Distribution along the Axial Direction
From a mechanics perspective, one could view the soft, 3D metamaterial as single piece of material to study its basic displacement responses under external force. Due to overall design characteris-

---

F1-3
F144
F2-4
(b)
(d)
The metamaterial design with soft network structure: (a) node placement, (b) sample designs, (c) normal-to-axia push, (d) Normal-to-surfaat push, (c) when pushed on the edge, and (f) twisted when pushed sideways.
Figure 1:
ties, we conveniently viewed the proposed design as a passive robotic finger to examine its unique stiffness performance. We developed simple test rig in Fig. 2(a) to measure the finger's radial stiffiess, and reorient the finger in Fig. 2(b) to measure its axial stiffness. Three simple design variations are used as sample designs for testing, including the F1-3 design in Fig. 2(c), F1-4 design in Fig. 2(e), and F2-4 design in Fig. 2(g). We designed these three samples to share the same height of 75 mm, the same 40 mm size at the bottom, the same number of three-layer designs in the middle, made from the same rubber material, and the same radius of the rubber bar in 2.5 mmm Small through-holes are also added to each finger sample for the ease of fixture on the test rig.
Based on previous observation of the finger's adaptive form response, we picked three different locations along its axial line marked in red, including the 2/5, 3/5, and 4/5 location of its total height, to perform the radial stiffness test. The results are plotted in Figs. 2(d). (f), and (h). To measure radial stiffness, soft fingers were mounted on a platform vertically and a bar horizontally pushed the fingers with mm,, 2mm, 3mm, 4mm, 5mm displacement on 2/5, 3/5, 4/5 location as showed in Fig. 2(a), the corresponding forces were recorded by VICTOR 100 digital force gauge. To measure the axial stiffness, soft fingers were mounted horizontally showed in Fig. 2(b) and a hook horizontally pulled the fingers with Imm, 2mm, 3mm, 4mm, 5mm displacement, and the forces were also recorded by the digital force gauge.
As the result shown in Figs. 2(d), (f), and (h), the relationship between force and displacement is linear in the same locations. Then first-order curve fitting method is used to calculate the line slope as stiffness. In some cases, the forces are null with small displacement (1mm and 2mm) as the digital force gauge is not sensitive.
2.3 Differential Stiffinss towards Omni-directiona Adaptation
significant design feature of the proposed finger is a network structure of tbree-dimension interac- tion, which is different from existing designs that are usually limited to two-dimensional adaptation [19, 20, 21, 22] making it soft, 3D metamaterial as a whole. shown Figs. 2(d), (f), and (h), results show that these finger samples exhibit consistent stiffness distribution in general, where the radial stiffness at 3/5, or the central location, is measured have the smallest stiffness. The radial stiffness is much increased at both 2/5 and 4/5 locations, which are of similar magnitude for each

---

(c)
Pull Direction
(e)
Pull Direction
(g)
Pull Direction
Finger
Digital Force Gauge
Push Position
Push Position
Push Position
Push
4/5
3/5
2/5
3/5
Push Bar
2/5
P rish
Push
Push Dingetion
Direction
Direction
F1-3
F144
F2-4
F1-3
F1-4
F2-4
(b)
(d)
pos 2/5
(f)
pos 2/5
(h)
Dos
Digital Force Gauge
105
3/5
4.5
008 3/5
00
3.0
Dos 4/5
4.0
pos 4/5
Dos 4:5
Foree N
Z.35
Force N
5
Tow Hooke
2.5
33.0
4
2.0
2.5
2.0
1.5
1.5
1.0
5 Displacement mm
2
3
2
4
2
4
Displacement mm
Displacement mm
Figure 2: Differential stiffness of the metamaterial that enables form adaptation: (a) radial com- pression experiment setup; (b) axial stretch experiment setup; (c), (e), (g) three fingers used in the experiments: F1-3, F1-4 and F2-4. The red arrow is the pull direction to measure axial stiffness in (b), the blue arrow is the push direction to measure radial stiffness in (a), and eyan dotted lines are four loading positions which are quintiles of the finger. (d), (f), and (h) are corresponding displacement-force curves of the fingers (c), (e), and (g).
design variation. On the other hand, when more nodes are used in the design, more rubber bars will be involved in the resultant finger, and overall radial stiffness becomes much larger.
The results are shown in the left side Fig. 3, which may be used to explain our soft finger's 's omni- directional form adaptation in the radial direction during the interaction. Here, we introduce the stiffinss ratio of the soft finger, measured by the ratio between the finger's s axial and radial stiffness. The soft finger generally exhibits relatively lower stiffness ratio near the central but higher readings near both ends, consistent among the three sample designs. When interacting with the environment in the radial direction, such stiffness ratio enables the finger easily deform towards the central part while portions near both ends deform much less. Furthermore, due to its axial symmetric design, such behavior can be easily observed in any radial direction of the finger, making omni-directional adaptive.
Finger curve
F2-4
Finger stiffness
1.3
finger
F1-3
postice
Tesh
0.50
FI-3
F2-4
Push
1.0
Sofinets Wine M0mm
0.40
0.9
0.35
F1
finger
F2-4
sample area
0.25
06
pos 3/5
05
Push
3/5
Positiin
Position mm
Figure 3: Stiffness distribution of the 6D finger the left and the vision finger the right. Left: the stiffness distribution of three vision fingers with horizontal loading. Right: the stiffness distribution of 6D finger with surface normal loading.
While the above test only measures reading on scattered points using simple force scales, we con- ducted more detailed measurement using Yaskawa's MotoMINI to conduct the same experiment automatically, In this experiment, we collected more refined measurements using ATI's nano25

---

6-axis force and torque sensor measuring at the finger's base while the robot pushes along the nor- mal direction to the finger surface along the whole length. A more slender design is tested with results plotted on the right of Fig. 3. The marked points correspond to the same 2/5, 3/5, and 4/5 locations of this particular soft finger, which exhibit similar distribution comparing to the F2-4 design on the left, but slightly skewed to the base side with higher stiffness at 2/5 location than the 4/5 location towards the tip. The results on the right side of Fig. indicate that the proposed soft finger may exhibit more complex behaviors in stiffness ratio along with its geometric form, making extremely challenging to model precisely using analytical method. In order to better leverage its omni-adaptive capability, will introduce two different data-driven methods to integrate the soft finger with machine learning, achieving similar performance in fundamental interactions as robot force control.
3 Learning Soft Robot Force Control
3.1 Sensing Differential Stiffness with Machine Vision
Rigid-soft interaction be hard model, it involves infinite degrees of freedom. To make the research feasible, we turn to the deep learning method at first sight. Deep learning constitutes modern technique for solving computer vision problems, with remarkable results and large potential. In order to make the best use of the data-driven method, we designed highly integrated vision- based soft finger, is shown in Fig. 4(a). With USB camera installed inside the structure of the soft finger, it can be easily recognized different interaction events based merely deformation images captured, such as Fig. ((b).(e) and (f). this highly integrated design, we fix the soft finger with a data during the interaction. With this vision integrated design, we attempt to solve the rigid-soft interaction problem by using a learning-based method.
F/T sensor ATI Nano25 [23], which is used for providing ground truth F/T
ArUco tag has been widely used in the computer vision and robotics community for its simplicity and low cost. In this paper, we also explored the possibility of learming the rigid-soft interaction of the soft finger by tracking the ArUco tag installed on the finger, shown in Fig. 4(e). The USB camera captured gray-scale images of 640 x 400 pixels at 120 frames per second (fps). Figs. 4(f), (g), and (h) show the camera captures of the ArUco code when the finger is without external force, with pushing on one side and with twisting around its vertical axis, respectively. The USB camera was connected tag placed at 26 mm away from the camera, the detection function in OpenCV [24] was able to achieve a successful detection rate 100% and the computational time cost is ms per image on average. By examine 15,000 images of the at the same pose, we found that the perturbation range of pose estimation [0.18, 0.06, 0.0008] mm for x, y, and [0.086, 0. 0.131, 0.004] degrees for roll, pitch, yaw. In the rest of the paper, always applied a moving average to the detected pose within a window length 5.
laptop with eight Intel processor running at 2.8GHz. With the ArUco
(s)
Soft Finger
b
(c)
Soft Finger
E
USB Cameea
USB Camera
Tag
E
ATI
Nano 25
Figure Vision integration with a camera mounted on the base: (a) Vision integration A; (b) default
camera view of A; (c) camera view of A during normal interaction; (d) camera view of A during twisting interaction; (e) Vision integration B; (f) default camera view of B; (g) camera view of B during normal interaction; (h) camera view of B during twisting interaction
5

---

3.2 Learning Rigid-Soft Interaction
During the interaction with the physical world, the soft finger will deform due to the contact force/torque. The idea of learning a neural network that maps sensor readings to 3D force esti- mation through labeled training data has been adopted in previous research [1]. The problem in this paper is to estimate the force and torque S' (Fx, Fy, Fz, Mx, My, Mz) E IR at the base of the soft sensor from the deformation of the metamaterial. Due to the complicated property of the metamaterial, we use an image of the internal structure to represent the deformation and use convolutional neural network to learn the mapping ft = F()).
In order to automatically collect the contact information with the minimal human invention, we build an automated data collection platform, as is shown in Fig. 5(a). To obtain high-quality con- trollable contact data, we choose Yaskawa Motomini [25], an industrial desktop robot arm capable of achieving 0.02 mm repeatability. We 3D print a T-Shape indenter and install it on the flange of the robot arm. As for the contact data measurement, we choose ATI Nano25, an expert industrial 6-axis F/T sensor, to record the contact force/torque at the soft finger base during the interaction. A commercialized USB camera captures the deformation image with resolution 640 x 480, which is easily installed inside the soft finger.
We use ROS [26] Yaskawa driver to control the movement of the robot arm. The indenter is com- manded to make indentation perpendicular the surface and move along the surface axis from bottom to tip of the soft finger. The waypoints along the surface axis are Imm spaced, and the robot will move along the surface's 's normal direction (pointed into the surface) and 3mm, at every waypoint. When the robot has moved to the target position, the camera and the F/T sensor will record pair of data points. We repeat the indentation test times for every surface. As result, we collected more than 2,500 data points and trained convolutional neural network.
depth of Imm, 2mm,
Convolutional Neural Network has become effective method to deal with computer vision prob- lems. The CNN structure is shown in Fig. 5(b). We deploy the residual blocks [27] as the feature extraction layers in the hope of capturing the deformation features of the soft finger during the inter- action. Then two fully connected layers are connected to the feature extraction layers, followed by a sigmoid activation function to get normalized predict results between [0. 1]. As we use the same feature extraction layers to make regression on multi-variables it is helpful to normalize the data before feeding it into the neural network. We use the CNN to directly predict the force along the a--axis and z-axis and torque along y-axis due to the contact condition. We use Adam Optimizer and train the network with mean squared error as a loss function.
We hold 10% collection data as the test set. The comparison result between visual force CNN prediction and ground truth is shown in Fig. 5(c). (d), and (c). The coefficient of determination R2a of the prediction along x-axis, z-axis, and torque along y-axis are listed in the figure. The figure shows that our visual force CNN is capable of making precise predictions, with R2 0.9 in each dimension. However, our model also shows its deficiency when making contact with objects other than the T-Shape indenter. The lack of generalization ability leads to bad deployment results when directly using its prediction as an F/T sensor. Therefore, we would like to explore more robust way to infer contact information based on internal vision sensing.
3.3 Tracking Interaction using 6D Pose
The deformations of the soft finger by external forces lead to the displacement of the tag from its original pose represented by (ōx, Py, 82, öroll, Spitch, 8yaw). Thus the soft finger might be a good joystick to teach the robot. When installed on the tool flange, the soft finger could achieve results similar to admittance control with a force/torque sensor. Considering that the soft finger is hardly affected by force along the z-axis, we designed a positional control strategy. We use the SPEEDL API from UR and control either the position of tool center point (TCP) or its orientation. In the former mode, the linear speed Ug and f. is proportional to dx, ōy while v. is proportional to Syaw. Similarly, in the latter mode, the rotational speed is proportional to ōz, dy while Vz. The 6D displacement of the tag is transformed to the frame of the robot base from that of the camera. Hence the interaction is very intuitive to the human user.
We conducted control experiments using a UR 10e robot arm, from which we can read the force/torque at the TCP directly from the controller. The task is to drag the TCP so that it follows particular path on a horizontal plane as shown in Fig. 6(a)-(d), namely printed circle and the

---

MAS
Surface Axis
Solt Finger
lavers
Motomini
312d
feolure 1
USB
FCs
I Shape Indenter
ATI
G fealure dhap
RegressuE
Namo15
(8)
(b)
Prodiction 6
Elediction 1 u200s)
200
150
y2
0.985
2
300
yd
0,987
0
Ground toth
Crnand Euth
(e)
Figure 5: Learning rigid-soft interaction on the surface normal: (a) experiment setup; (b) network design; (c) Force prediction result along x-axis on test (d) Force prediction result along z-axis on test set; (e) Torque prediction result along y-axis on test set.
abbreviation of Conference on Robot Learning. We record the positions of the TCP while we taught the robot and plotted the resulting paths using a force-torque sensor and the soft finger. The soft finger has demonstrated excellent experience of human-robot interaction and was able to complete the tasks well.
@
(b)
Fx(N)
1.05
Ma(Nm) MyNMm) Ms(Nm) Fx() Fy())
10
0.5
(
-0.5
CORL CORL
+0.5
0.5
-0.
1000
3000
4000
Time
Figure 6: Tracking interactions by visual estimation of the internal 6D pose: (a)(c) paths completed using the soft finger; (b)(d) paths completed by teaching the robot using force/torque sensor on UR10e; (e) real-time force and torque predicted by the learned model in red against those obtained from the UR 10e's controller in black.
A further hypothesis is that the 6D displacement of the tag could well represent the deformation of the soft finger under external force. Hence mapping from such displacement to the force could be learned using the machine learning method. We installed a soft finger the tool flange of the

---

UR10e. We wrote a program to automatically record the images captured by the USB camera and the force/torque values read from the robot controller while we push and twist the soft fingers by hand. With the high fps camera, we collected total of 30000 images and their corresponding force/torque values in a few minutes. We used single regression model of three hidden layers with 200, 100, and 6 neurons, respectively, and the input is the 6D displacement of the tag, and the output is the 6l) force and torque. The loss function is mean squared loss, and the optimizer is Adam with a learning rate of 0.001. The regression converged quickly after few epochs.
To test whether the learned model can be transferred to new sensor of the same design, we have fabricated two fingers of the same design, material, and fabrication methods. One was used to collect the training data as described above and was later replaced by the other finger to test the performance. The standard deviations of the predicted signals (Fx, Fy, F2, Mr, My. 2) of the soft sensor measured over 100 seconds are (0.009, 0.034, 0. N.007) and (0.0013, 0.0009, 0.0010) m. The same measurement for UR 10e are (0.323, 0.431, 0. 0.253) N and (0.0123, 0.0107, N.0058)NN... m. The noise level of the soft sensor is around 10 times lower than that of the UR10e's. s. Fig. 6(e) shows the real-time force and torque predicted by the learned model in red, and the values obtained from the robot controller in black. The model can give a fairly good prediction even with many discrepancies between the prediction and the read values from the controller.
4
Final Remarks
In this paper, we explored the visual integration of soft, 3D metamaterial with an onni-directional adaption by adding monocular camera the design base adopt a data-driven method for human- robot interaction. We systematically investigated the metamaterial's design characteristics, which feature radial stiffness ratio with differential distribution along its axial direction, resulting in an omni umni-directional adaptation that is challenging to model, yet attractive to interact with. We proposed two different design enhancements the original finger design by simply adding a camera at the base or fix another fiducial marker inside. We built several learning models to investigate the mapping between the finger mechanics from vision and the force and torque response from the interaction. Our results show that it could become promising direction to use data-driven learning method to leverage the modeling challenge of soft robots for advanced robot control and interaction, where a potential future for soft robot force control may be of interest to researchers seeking more adaptive, versatile, and low-cost sensing and interaction of their robotic system.
As the soft sensor senses through visual signals, its output frequency is determined by the frequency of the camera and the inference frequency of the prediction model. In our experiment, the frequency is currently limited by the frequency of the camera, which is 120 Hz and is already higher than most previous research tactile sensors employing camera [28]. This is comparable to the commercial product FT 300 from Robotiq, which is 100 Hz. As the computation time of ArUco tag detection ms on average, the upper bound could further increased to 333 Hz with higher speed camera. In general, tactile sensor employing camera suffers from the relatively low frame rate and higher computational cost compared with other conventional sensor. However, it made the proposed soft sensor to be more accessible and easily assembled with much lower cost. The total cost of such soft finger with sensing capability is lower than 100 USD.
As reviewed by [28), most of the previous research in tactile sensors employing camera converted the physical contact into light signal based on some kind of micro or local deformation of the contacting point and the camera was placed within an controlled environment with stable lighting devices such as LED. One of the major difference of our proposed soft sensor is that the physical contact light conversion is based on large deformation of the soft metamaterial. In order to investigate the soft metamaterial alone, we kept it bare and did the experiment in daily lighting condition in the lab without any extra lighting devices. The stability could be harmed if the lighting condition become weak, but can be engineeringly improved with low-cost artificial lighting when necessary.
Due to the limited precision of the force/torque sensor build in UR 10 e-series, it's not worthwhile to compare the model prediction the values read from controller. We plan collect more data and use force/torque sensors with higher precision to better characterise the soft fingers during in- teraction. Looking for a potential low-cost replacement of the industrial force/torque sensor, we will further improve the design of the soft sensors to overcome some of the current limitations and explore promising application scenarios.
8

---

Acknowledgments
The authors would like to acknowledge the constructive comments provided by the reviewers and editors. This work was jointly supported by National Science Foundation of China (No. 51905252), Shenzhen Institute of Artificial Intelligence and Robotics for Society Open Project (No. AC01202005003), Shenzhen Long-term Support for Higher Education, and AncoraSpring. Inc.
References
[1] B. Sundaralingam, A. S. Lambert, A. Handa, Boots, Hermans, S. Birchfield, N. Ratliff, and D. Fox. Robust learning of tactile force estimation through robot interaction. In 2019 International Conference on Robotics and Automation (ICRA), pages 9035-9042, IEEE, 2019.
[2] Y. S. Narang, K. Van Wyk, A. Mousavian, and D. Fox. Interpreting and predicting tactile signals via physics-based and data-driven framework. arXiv preprini arXiv:2006.03777, 2020.
[3] K. Gutierrez and V. J. Santos. Perception of tactile directionality via artificial fingerpad de- formation and convolutional neural networks. IEEE Transactions on Haptics, 13(4):831-839 2020.
[4] D. E. Whitney, Historical perspective and state of the in robot force control. The Interna- tional Journal of Robotics Research, (4):3-14, 1987.
[5] P. Xu, A. Mishra, H. Bai, C. Aubin, L. Zullo, and R. Shepherd. Optical lace for synthetic afferent neural networks. Science Robotics, 4(34):eaaw6304, 2019.
[6] Y. Yan, Z. Hu, Z. Yang, W. Yuan, C. Song. J. Pan, and Y. Shen. Soft magnetic skin for super- resolution tactile sensing with force self self-decou Science Robotics, 6(51))eab,880,, 2021.
[7] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolttion robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
[8] E. Donlon, Dong, M. Liu, J. Li, Adelson, and A. Rodriguez. Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger. In 2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 1927-1934. IEEE, 2018.
[9] B. Wang, W. Guo, S. Feng, H. Yi, Wan, and C. Song. Volumetrically enhanced soft actuator with proprioceptive sensing. IEEE Robotics and Automation Letters, 6(3):5284- 5291, 2021.
[10] D. Kim, S.-H. Kim, Kim, B. Kang. M. Lee, W. Park, Ku, D. Kim, J. Kwon, H. Lee, J. Bae, Y.-L. Park, K.-J, Cho, and S. Jo. Review of machine learning methods in soft robotics. PLOS ONE, 2(16):1-244 2021.
[11] M. Kadic, G. W. Milton, M. van Hecke, and M. Wegener. 3d metamaterials. Nature Reviews Physics, (39:119-210, 2019.
[12] Y. Liu and X. Zhang. Metamaterials: new frontier of science and technology. Chemical Society Reviews, 40(5):2494 2507, 2011.
[13] J. Weng, Y. Ding, C. Hu, X.-F. Zhu, B. Liang, J. Yang, and J. Cheng. Meta-neural-networkk for real-time and passive deep-learning-basee object recognition. Nature Communications, 1(11): 6309, 2020.
[14] Jiao and A. H. Alavi. Artificial intelligenc--enabled smart mechanical metamaterials: advent and future trends. International Materials Reviews, 6(66):365-39) 393, 2021.
[15] F. Wan, H. Wang, J. Wu, Y. Liu, and C. Song. A reconfigurable design for omni-adaptive grasp learning. IEEE Robotics and Automation Letters, 420 2020.
[16] Z. Yang, S. Ge, F. Wan, Y. Liu, and C. Song. Scalable tactile sensing for an omni-adaptive soft robot finger. In 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), pages 572-577. IEFE, may 2020.

---

[17] L. Yang, F. Wan, H. Wang, X. Liu, Y. Liu, J. Pan, and C. Song. Rigid-soft interactive learning for robust grasping. IEEE Roboties and Automation Letters. 1227 2020.
[18] L. Yang, X. Han, W. Guo, F. Wan, J. Pan, and C. Song. Learning-bar optoelectronically innervated tactile finger for rigid-soft interactive grasping. IEEE Robotics and Automation Letters, 6(2):3817-3824 2021.
[19] Festo, 2009, BionicTripod with FinGripper, available at: wtt..////ssssspppt/tr/rieeeeeeee
pdf Google Scholar.
[20] W. Crooks, G. Vukasin, M. O'Sullivan, W. Messner, and C. Rogers. Fin ray@ effect inspired soft robotic gripper: From the robosoft grand challenge toward optimization. Frontiers in Robotics and AI, 3:70, 2016. ISSN 2296-9144. No::10.338/froob... 2tt.:/rooo.200..0.00 201.000070. URL https: h/www.frontii.
wttgarttcle////eoooooooooo...eet 3389/3309bbbb 2016. 00070.
[21] W. Crooks, S. Rozen-Levy, B. Trimmer, C. Rogers, and W. Messner. Passive gripper inspired by manduca sexta and the fin ray effect. International Journal of Advanced Robotic Systems, 1444):17298814177215555 2017. 101110.1777177998844777220 10. 10.1177/11298881177711
URL https: :ttpps://door htto:/doooooooo
[22] C. Basson and G. Bright. Geometric conformity study of a fin ray gripper utilizing ac- tive haptic control. In 20/9 IEEE 15th International Conference Control and Automation (ICCA), pages 713-718, 2019.
[23] ATI, Nano25, available at: stttps:/wwwx ati- iiii-ia.ee httaaaa...c//prruuuttt////tttaaa aspx? id=Nano25.
[24] G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
[25] Yaskawa, Motomini, available at: https: www.yyaawwa....oppooottttt/ooooa mottom/
[26] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. N. al. Ros: an open-source robot operating system. In ICRA workshop on open source software, volume 3, page 5. Kobe, Japan, 2009.
[27] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, 2015.
[28] K. Shimonomura. Tactile image sensors employing camera: review. Sensors, 19(18):3933, 2019.
10


Reviews:
[{"review_id": "_0ut6BetFtc", "reviewer": "Reviewer_h8MY", "summary": "The paper develops a soft finger and leverages learning to estimate force/displacement from visual observation of deformations during interaction with external objects.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is leveraging learning to enable force estimation from deformations of a 3d printed finger leveraging a camera. This could enable low cost fingertips for use in the industry where the fingertips can be swapped out with minimal cost. \n2. Designing these fingertips to have 6d force/torque estimation is very interesting and leveraging learning to go around modeling the deformations has great benefits as shown by the paper.\n3. The teleop application shown with the proposed method is very cool and shows a direct application of the approach.\n\n\nWeakness:\n1. The paper learns a mapping from finger deformation to 6D force/torque. However the accuracy of the predictions is not validated. Plotting the error in predictions both in magnitude and direction would be helpful. Previous force estimation methods have reported errors on test dataset to help compare across methods [biotac_force].\n2. Does the learned model transfer to a new sensor of the same design? Discussing this question with some experiments is needed. \n3. The paper does not cite any existing work on tactile force estimation. Specifically, the idea of collecting data leveraging robots and learning a NN is not novel, it has been used to estimate force and deformation fields in previous works[biotac_force, biotac_physics].\n4. Why are the predictions very noisy? Is there any filtering that is done post prediction to enable smooth teleop? \n5. The paper doesn’t show what happens when force is applied in non-normal directions, i.e. between parallel (shear) and normal angles. How does the accuracy change when force is applied in these directions? While estimating shear force is not necessary, having accurate force angle prediction is important for many manipulation tasks[biotac_directionality].  Showing some plots when force is applied in these directions is required to better understand the limitations of the work.\n6. Can the force estimated be used for manipulation tasks such as gentle object placement? [biotac_force]\n\n\nReferences:\n\n[biotac_force] Sundaralingam, Balakumar, Alexander Sasha Lambert, Ankur Handa, Byron Boots, Tucker Hermans, Stan Birchfield, Nathan Ratliff, and Dieter Fox. \"Robust learning of tactile force estimation through robot interaction.\" In 2019 International Conference on Robotics and Automation (ICRA), pp. 9035-9042. IEEE, 2019.\n\n[biotac_physics] Narang, Yashraj S., Karl Van Wyk, Arsalan Mousavian, and Dieter Fox. \"Interpreting and Predicting Tactile Signals via a Physics-Based and Data-Driven Framework.\"\n\n[biotac_directionality] Gutierrez K, Santos VJ. Perception of Tactile Directionality via Artificial Fingerpad Deformation and Convolutional Neural Networks. IEEE Trans Haptics. 2020 Oct-Dec;13(4):831-839. doi: 10.1109/TOH.2020.2975555. Epub 2020 Dec 25. PMID: 32092014.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper does not introduce any novel learning/data collection methods. But leverages learning to enable force estimation from hard to model deformations of a fingertip. There could be several applications of the proposed solution. Addressing the following is required to improve the recommendation: \n1. Reporting magnitude and directional accuracy.\n2. Analyzing transfer across other fingertips of the same design.\n3. Analyzing accuracy of force estimation in non-normal directions.\n4. Showing a force controlled task such as grasped object placement based on detection of object contact with table.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Issues are listed in recommendation.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "J6Uwk7gHXa8", "reviewer": "Reviewer_avKF", "summary": "The paper applies a deep learning approach utilizing convolutional neural networks to the problem of extracting torque measurements from a soft structure. This structure, having a pyramid-like form, is meant to be attached to the end-effector of a robot. Images and videos show an example where it is mounted on a UR 10 robot arm. The general idea is to record camera images of the internal deformation of the structure from its base and then using a supervised learning approach to learn a neural network predicting the external force. The motivation for this is the idea that the structure can be used as a 'soft finger' on a robot to gently guide it for demonstrating/recording movement trajectories.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although not entirely novel (see below), the idea of extracting force measurements from visual observations is a good line of enquiry for this paper. Furthermore, the general setup of the experimental testbed seems rigorous. The video is of very good quality.\n\nHowever, I have major concerns regarding the general learning setup and its motivation as it is described in the paper. \n- The general learning problem is not very well formalized: What exactly is the learning problem the paper tries to tackle? Just as an example, the paper does not mention that the learning setup is supervised at least once. The greater question is, why is the learning problem supervised? Why not unsupervised or using reinforcement learning? Furthermore, the ATI Nano25 sensor appears smaller than the camera box at the bottom of the finger. Why not simply attach the ATINano25 sensor instead of a camera? What extra information would the camera provide, and why would we be unable to use an F/T sensor at the base?\n- The estimated force/torque data is utilized to be fed back into the controller of the UR10 in task-space, controlling the position of its end-effector. Why can this not be solved with said additional force sensor at the base of the finger or a more sensitive (robot) controller?  This setup would not require deep learning at all.\n- Why is deep learning the algorithm of choice? Why not use the information from the tag extracted with OpenCV to train an SVM or another regression algorithm? Figure 5 shows only the predictions of the used neural network, some other learning algorithm as baseline would be expected. Or an explanation of why other algorithms are not applicable.\n- I cannot entirely agree with the conclusion that the proposed setup would allow other researchers a more versatile or low-cost setup: Assuming reasonable variance in manufacturing or 3D-printing the proposed structure, every lab would have to carry out the calibration method described, i.e. collecting a supervised data set with force measurements and images showing the deformation. This process would also be required to be repeated every time a new finger/structure is produced. \n- The paper introduced the soft structure as a 3D metamaterial. As the introduction of the paper already mentions, a meta-material is usually a composite material, i.e. a material made up of at least two or more materials. It is not clear if this is the case here, or if simply the structure of the elastic material (in the form of a pyramide like tubing) determines alone its properties. \n- For me, the results presented in Figure 6 (b) (UR 10 force sensor) look considerably more natural and better than (d) (proposed camera-based approach).\n\nA more interesting approach would have been to learn the required force/torque estimated in a reinforcement learning setup, in the absence of any additional sensory equipment, by letting the robot arm interact in some fashion with the environment. This would also strengthen the applicability of the paper for a robot learning conference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "From the point of robot learning, it is not clear what the exact contribution of the paper is. The described learning setup is rather weak, and the principal problem could be solved by exchanging the camera with a force/torque sensor and a more sensitive robot control algorithm, which would not require learning at all. Furthermore, the approach is limited to setups where it is possible to acquire a supervised data set by the use of force/torque sensors. A more exciting approach could be an autonomous learning process in which the robot arm learns the direct mapping from captured structural deformation to motor commands. This would also allow the operator to apply the proposed finger structure to any robot, not just ones with accurate kinematic modelling in the absence of any force/torque sensors.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Figure 5: The graphs should show a grid-pattern to allow the viewer to evaluate the deviation between predicted and actual force. Furthermore, both x- and y-axis should have the same range.\n- Section 3.2 should be restructured and introduce the exact learning problem in a formal way. \n- Explain what the exact problem the paper is trying to solve is. Why do we need deep learning for it? How can it be made applicable to the outside of your lab and to other researchers who might want to use the proposed finger structure but do not have access to the testbed used to collect the supervised data?\n- Is the proposed approach using CNN networks really the only approach capable of solving this problem? For example, how does the proposed algorithm using a camera compares against directly attaching a force/torque sensor to the base of the finger structure?\n- Fig. 6 should show in (a-d) the reference trajectory to follow.\n- Lastly, why use a supervised learning approach + a controller in task-space? Why not directly learn the motor commands given the deformation in the finger structure? \n- The paper could discuss further camera-based approaches for manipulation such as \"OmniTact: A Multi-Directional High-Resolution Touch Sensor\"\n- The paper would also benefit from a better discussion about the potential applications of this structure, especially regarding the prospect of using it for human demonstrations in the robot learning community."}}, {"review_id": "86EUBzD_GT", "reviewer": "Reviewer_Wc5U", "summary": "The paper explored the use of soft material for robot force sensing and control. The main contribution is the introduction and analysis of a cone-shaped metamaterial, and two vision-based approaches to infer the applied force on the material based on deformation. The proposed system is evaluated on a soft robot force control task.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of leveraging soft materials and computer vision for robot force control is interesting and reasonable.\n2. The proposed metamaterial structure is concrete and carefully analyzed.\n3. Two different methods were evaluated to learn the force profile of the soft material under different forces and are both shown to be effective, though one has some generalization issue.\n4. Preliminary real-world experiments were conducted to demonstrate the usefulness of the proposed idea.\n\nWeaknesses:\n1. Limitations of the proposed method are not well discussed. For example, what’s the highest frequency that the proposed system can run at? A force-torque sensor can usually run at very high frequency but a camera may have limited frequency. Another potential limitation is that the system would be more sensitive to lighting conditions and also introduce a higher cost of computation in exchange for lower cost in hardware. Some discussion along these lines would be helpful.\n2. The proposed system didn’t demonstrate superior task performance in the evaluated task: in Figure 6, it seems that with force-torque sensors the resulting trajectory is smoother.\n3. For the AR-tag approach, is it able to infer pulling or push forces on the material? For example if the robot hits a wall vertically, it would be important to properly detect that.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the idea presented in this paper is interesting and promising in my knowledge. The design of the meta-material and the presented learning approach to infer the force and torque readings are solid. On the other hand, some more discussions are needed and a better experiment setup would make the paper even stronger and more interesting. For example, if one can build a two-finger gripper out of the proposed method, it’ll be able to sense forces per-finger. This is something difficult to achieve with existing methods and can potentially improve manipulation performance, which would better support the value of the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see weaknesses.\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "0XqKIFA8KQ_", "reviewer": "Reviewer_mjxa", "summary": "This paper explores the feasibility of learning robot force control and interaction using soft metamaterial and machine vision.\n\nThe paper explores two methods: directly learning the force and torque response from raw images of the metamaterial’s internal deformation, and indirectly estimating the 6D force and torque using a neural network by visually tracking the 6D pose of a marker fixed inside the 3D metamaterial. The paper finally integrates the two methods and achieves similar force feedback and control interactions in simple tasks.\n\nThe results show that the learning method holds the potential to support the concept of soft robot force control, providing an intuitive interface at a low cost for robotic systems, generating comparable and capable performances against classical force and torque sensors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have no knowledge in this domain before. I cannot evaluate the novelty and technical contribution of the paper correctly. I only make sure the claims in the paper are correct.\n\nStrength:\n\nThe paper has several contributions: \n-- The investigation of a soft, 3D metamaterial with differential stiffness for passive form adaptation in omni-directions in the radial plane;\n-- A direct vision-based sensory integration of the 3D metamaterial for estimating the force and torque using a modified resnet;\n-- The design integration of a marker inside the omni-adaptive metamaterial to facilitate efficient, accurate, and real-time 6D force and torque using vision-based pose tracking;\n-- System integration towards soft robot force control with comparable performance against commercial force and torque sensors with much-reduced hardware cost and complexity.\n\nAll the contributions are clearly stated and empirically demonstrated in the paper.\n\nWeakness:\n\nI do not see explicit weakness in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I cannot evaluate the paper correctly so I give a rough evaluation of the paper. I will discuss with other reviewer about the final recommendation for this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "No issue.", "reviewer_expertise": "Poor: Limited knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes achieving soft robot force control using a 3D flexible structure and learning estimates of force-torques by using vision, potentially decreasing the cost of robotic systems. In general, the paper has been considered of having a good impact, originality, and technical quality.

Amongst others, reviewers have raised issues regarding deficiencies in the related work, confusion concerning effectiveness in the estimation of forces in certain directions (vertically hitting a wall), and whether the performance against the conventional force-torque sensor is truly comparable. Additionally, it seems that a bit more motivation and more details for the CNN have been missed. The authors are encouraged to discuss the raised points with the reviewers during the rebuttal.

Post rebuttal ==============

The paper has been gone through extensive discussions between authors and reviewers. The overall scores have been increased during the rebuttal as the authors clarified most of the concerns. As a final result, we observe an improved final manuscript.


Decision: Accept (Poster)