1Insight: A Vision-based Haptic Sensor That
Provides Rich Force Information about Contacts
Huanbo Sun1 , Katherine J. Kuchenbecker2 , and Georg Martius1
1 Autonomous Learning Group2 Haptic Intelligence Department
Max Planck Institute for Intelligent Systems, Germany
Email: huanbo.sun|georg.martius@tuebingen.mpg.de; kjk@is.mpg.de

Robots need detailed haptic sensing that covers their surfaces in order to learn effective behaviors in unstructuredenvironments. However, state-of-the-art sensors tend to focus
on improving precision and sensitivity, increasing taxel density,or enlarging the sensed area, rather than prioritizing systemrobustness and the usability of the sensed haptic information.
We thus present a robust, soft, low-cost, vision-based, thumbsized 3D haptic sensor named Insight: it continually supplies
the host robot with a directional force-distribution map over its
entire conical sensing surface. Insight uses an internal monocularcamera, photometric stereo, and structured light to detect the 3D
deformation of the easily replaceable flexible outer shell, which ismolded in a single layer over a stiff frame to guarantee sensitivity,robustness, and a soft contact surface. The force information
is inferred by a deep network (ResNet) that maps images tothe spatial distribution of 3D contact force (normal and shear),including numerous distinct contacts with widely varying contact
area. Extensive experiments show that Insight has an overall
spatial resolution of 0.4 mm, force magnitude accuracy around0.03 N, and force direction accuracy around 5 degrees over arange of 0.03–2 N. The presented hardware and software designconcepts can be extended to achieve robust and usable tactile
sensing on a wide variety of robot parts with different shapes
and sensing requirements.

1. I NTRODUCTION
High-fidelity haptic sensors with 3D sensing surfaces areneeded to advance dexterous robotic manipulation. Comparedto the 2D flat geometry of most state-of-the-art tactile sensors,
a 3D sensing surface can enable robots to feel contact on all
sides of their bodies, like human skin, which will let themwork better in messy real-world environments [1, 2]. Beyond
the shape of their sensing region, currently available sensors
have other limitations that prevent their widespread applicationin the field of robotics, including low spatial accuracy, aninability to sense shear forces, complex fabrication processes,
and/or low durability for long-term use. We sought to improve
the situation by designing a durable high-resolution haptic
sensor that has a 3D sensing surface and is easy to manufacture [3].
Among different sensor design options, vision-based haptic
sensors are a new family of solutions, typically using aninternal camera that views the soft contact surface from within[4]. The cameras available nowadays have high resolution,
good quality, high robustness, and a low price. They cancapture simultaneous data across a large area, which is very
useful for creating a tactile sensor for robots. Moreover,Fig. 1. We design a soft thumb-sized vision-based haptic sensor that gives
detailed force distribution information over its 3D sensing surface.

advanced image-processing techniques offer references for an
accurate way to transform the image data into tactile contact
measurements.
In this paper, we design a vision-based haptic sensor andpropose a machine-learning method to offer a 3D directional
force distribution map over a 3D conical thumb-sized sensing
surface, as shown in Fig. 1. We detail the sensor’s working
mechanisms in Fig. 2. The sensor comprises a miniaturecamera, an LED ring, and a soft-stiff hybrid surface. The
hybrid structure of Insight’s soft elastomer shell with a stiff
metal skeleton inside ensures high sensitivity and robustness.
Colored LED light is projected on the inner side of the opaquesensing surface, and the camera looks through the ring of lights
to see the inner reflective surface. When an object contacts
the outside of the sensor, the camera sees changes causedby the deformation. A trained machine-learning model runs
to interpret each camera images as a force map. The force
map describes the directional force distribution over the entire
3D sensing surface, i.e., 3D force vectors at a fine grid ofpoints distributed all across the sensor. A testbed with 5-DoF
actuation and 3-axis force measurement is built to carry a four
millimeter spherical indenter to probe the sensing surface andcollect the needed data to train the machine-learning model.2A

Force Prediction

Sensing
Contact Image-3.4 mN

Skeleton

LED Ring

Collimator

Supporter

Fisheye Lens

Image DAQConnector

BCameraElastomer

Force Visualization-4.2 mN+17.6 mN

ResN
et-4.3mN-21.9 mN
+5.7 mN

Reference Image
fxfyfzFig. 2. A: Insight has a hybrid mechanical structure of a soft elastomer enclosing a stiff metal skeleton for both sensing light contact and sustaining
high-impact force. An LED ring and a collimator are combined to create structured light for detecting 3D deformations of the surface from a single 2D image.
B: A trained machine-learning model takes images from the camera and a reference image to estimate the force distribution all over the surface. Each pixel inthe force prediction map has three values that indicate the force strength in three directions. In the force visualization on the right, each point (correspondingto one pixel in the force prediction) shows the force distribution of the contact in both normal and shear directions.

We quantitatively evaluate the sensor performance over aforce range of 0.03–2 N. It can predict single contact by
the four millimeter spherical indenter with an overall spatial
precision (RMSE) of 0.4 mm and force magnitude around0.03 N. It can tell the detailed directions (normal and shear)of the contact force on the surface with a direction accuracy
around 5 degrees. The sensor can detect multiple contacts
simultaneously and detail the contact shape when objects
are pressed into the finger-nail zone. Finally, the sensor
can precisely predict its own orientation (accuracy around 2degrees) relative to gravity by visually observing the small
gravity-induced deformations of the over-molded elastomer
with ridges.

Our sensor design concept can be applied and extended toa wide variety of robot body parts with different shapes andprecision requirements. We have open-sourced all the design
files and code [5], and we have also provided ideas on how toadjust Insight’s design parameters for other applications, suchas the field of view of the camera, the arrangement of the lightsources and the composition of the elastomer [3].

R EFERENCES[1] J. A. Fishel and G. E. Loeb, “Sensing tactile microvibrations with the BioTac — Comparison with humansensitivity,” in Proceedings of the IEEE International
Conference on Biomedical Robotics and Biomechatronics(BioRob), 2012, pp. 1122–1127.
[2] B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone,
M. E. Giannaccini, J. Rossiter, and N. F. Lepora, “TheTacTip Family: Soft Optical Tactile Sensors with 3DPrinted Biomimetic Morphologies,” Soft Robotics, vol. 5,no. 2, pp. 216–227, Apr 2018.
[3] H. Sun, K. J. Kuchenbecker, and G. Martius, “A soft
thumb-sized vision-based sensor with accurate all-roundforce perception,” Nature Machine Intelligence, vol. 4,Feb. 2022. [Online]. Available: https://rdcu.be/cHCl9[4] W. Yuan, S. Dong, and E. H. Adelson, “GelSight: HighResolution Robot Tactile Sensors for Estimating Geometry
and Force,” Sensors, vol. 17, no. 12, p. 2762, Nov. 2017.
[5] H. Sun, “Insight: A haptic sensor powered by visionand machine learning,” https://github.com/Huanbo-Sun/
Haptics-for-Robots-Insight, 2022.