IEEJ International Workshop on Sensing, Actuation, Motion Control, and Optimization

FingerVision for Tactile Behaviors, Manipulation,and Haptic Feedback Teleoperation
Akihiko Yamaguchi∗a)Non-member

This paper introduces a vision-based tactile sensor FingerVision in order to attract haptics researchers to use FingerVision. We describe the overview of FingerVision and its applications including tactile behaviors and robot
manipulation. We also discuss the use of FingerVision for teleoperation with haptic feedback under learning-fromdemonstration context.
Keywords: FingerVision, Tactile sensing, Tactile behaviors, Robot manipulation, Haptic feedback teleoperation

1.

Introduction

This paper introduces a vision-based tactile sensor FingerVision and its applications in order to attract haptics researchers to use FingerVision. FingerVision was proposedin (1) by Yamaguchi and Atkeson. Its concept is combining
proximity vision and tactile sensing. Unlike other visionbased tactile sensors such as (2)∼(15), the cameras inside
FingerVision can see the outside of the sensor through theskin, which increases the sensing modality. The structure ofFingerVision is simple, consisting of elastic and transparent
skin, frame, and cameras. Markers are attached on the skinsurface for detecting skin deformation. The features of FingerVision are summarized as follows:(1) Multimodal: It can sense force distribution, slip, object
pose, texture, and other information obtained from proximity vision (computer vision for nearby objects). The
remarks are that: (1.1) Slip can be detected regardless thereactive force from objects. It can sense slippage evenwhen the object is too light to sense force (e.g. origami
crane). (1.2) Cameras can sense objects before collision.
With this feature, we can create safe interactive robots
that are aware of nearby humans.
(2) Easy to manufacture: Because of its simple structure,
its fabrication is easy.
(3) Low cost: The most expensive component is the camera. Other components are inexpensive.
(4) Physically strong: External force applies to the skin andframe, and does not reach the camera. Thus it is physically strong. Even if the skin is damaged, replacing it isinexpensive.
(5) The sensing elements (cameras) do not have to cover thewhole surface: Using wide-angle lenses (fisheye lenses),
we can make the camera allocations sparse.
(6) Processing software could be common, which enables
efficient development.
(7) Sensor parameters are adjustable: We can adjust the
dynamic range of force (hardness and thickness of theFig. 1. Conceptual design of FingerVision (a, b) and its
prototype installed on Baxter electric parallel gripper (c).

skin), size (small cameras miniaturize the sensor size),spatial resolution (camera resolution, marker allocation,etc.), and temporal resolution (high speed cameras forhigh FPS).
(8) Other types of sensors can be used: For example thermal cameras.
In (16), we demonstrated that FingerVision makes it easy
to program some tactile behaviors such as gentle grasp, holding without slip, natural handover, and in-hand manipulation.
Especially because of its sensitivity of slip, it was possible to
make a grasp adaptive controller which adjusts the grasp to
avoid slip during picking up an object (17) . With this controller,
the robot with a parallel gripper where the FingerVision sensors are installed could pick up a range of deformable andfragile objects, such as vegetables, fruits, origami arts, raw
eggs, and potato chips.
We encourage many researchers reproduce FingerVision
for their own devices. For this purpose, FingerVision is open
source (18) .
In the rest of this paper, we summarize the sensing technology of FingerVision, its applications to tactile behaviors
and manipulation, and we discuss the use of FingerVision
for teleoperation with haptic feedback under learning-fromdemonstration context.
2.

FingerVision is a vision-based tactile sensor consisting ofelastic and transparent skin made with silicone, frame madewith 3D printed support and acrylic, and cameras. Fig. 1
shows the conceptual structure. Markers are placed on thesurface of the skin for detecting skin deformation to estimatea) Correspondence to: info@akihikoy.net∗
Graduate School of Information Sciences, Tohoku University
6-6-01 Aramaki Aza Aoba, Aoba-ku, Sendai 980-8579, Japanc 2018 The Institute of Electrical Engineers of Japan.

FingerVision1

FingerVision for Tactile Behaviors, Manipulation, and Haptic Feedback Teleoperation (Akihiko Yamaguchi)

Fig. 2. Example of sensing when pushing the sensor by
finger. Force distribution, slip, and object (finger) aredetected.

external force distribution. In the prototype of FingerVision(Fig. 1(c)), we use Silicones Inc. XP-565 that has A-16 Shore
hardness as silicone, black micro plastic beads of 1 mm diameter for markers, and ELP Co. USBFHD01M-L180 USB
camera with fisheye lens. The thickness of silicone is 4 mm,
the markers are allocated on 5 mm grid, and the thicknessof acrylic is 2 mm. The camera stream is taken at 320x240,
30-60 FPS.
The raw measurement from FingerVision is image sequence as usual cameras. We use several different computer
vision methods for estimating force distribution, slip, and object pose. Estimating force distribution is achieved by tracking the markers whose movement tells us the deformation of
the silicone skin. Estimating slip and object pose are doneby directly analyzing the camera stream. We refer to suchcomputer vision for nearby objects as proximity vision.
2.1 Force Estimation By Marker Tracking
Fortracking the markers placed on the surface of the elastic
skin, we use a blob tracking method. It consists of two
processes: calibration to detect initial marker positions, andtracking the marker displacements. In the both processes,
the camera image is rectified for compensating the distortion caused by fisheye lens, and then converted to a black
and white image by thresholding black color as the markers are black. In the calibration, we cover the sensor withwhite sheet in order to avoid detecting noise from background. A blob detection method implemented in OpenCV(cv::SimpleBlobDetector) is used. The calibration takes
less than 1 second. Marker tracking is done independently
per marker. We assume a small region around a marker at its
previous position, and apply the same blob detection methodof OpenCV. If the marker movement is unexpectedly large,
we reject the result since it would be noise. We also compare the size of blob to distinguish the noise. The obtained
marker displacements are converted to 3-dimensional force
estimates. Refer to (16) for the details of force estimation.
Fig. 2 shows an example of marker tracking where the redbars highlight the marker movement.
2.2 Proximity Vision
We directly process the camera stream for proximity vision. Although we could use
many of computer vision algorithms for proximity vision
which would increase the sensing modality, we implement
two algorithms: nearby-object detection and slip estimation.
The object detection is useful to estimate the pose (position and orientation) and size of grasped object or nearby
object, and is also used to distinguish the background andobject movement. Slip is estimated by a background subtraction method with the mask of detected object. We also
considered optical flow, but background subtraction was bet-

Fig. 3. Baxter system with two grippers where FingerVision is installed on each finger. Raspberry Pi is used
to stream the camera data to Ethernet network.

Fig. 5. Examples of grasp adaptation control.

ter in some cases where the object does not have sufficient
texture. In the calibration stage, we build the backgroundmodel. An object model is we adaptively and dynamicallyconstructed. Both the background and the object models
are represented as color histograms. As the backgroundsubtraction method, we use an implementation of OpenCV(cv::BackgroundSubtractorMOG2). With this implementation, the strength of slip at each pixel is estimated which
does not have the direction of slip. Refer to (16) for the details of slip and object detection. Fig. 2 shows an example ofdetecting nearby-object and slip.
3.

Tactile Behaviors and Manipulation

We briefly describe the tactile behaviors and manipulation
with FingerVision to demonstrate the variety of its applications. Here we assume that the FingerVision sensors are installed on parallel grippers mounted on a dual-arm robot Baxter (Fig. 3). As the gripper, we use a Baxter electric parallel
gripper and a Robotiq 2-finger adaptive robot gripper-85. We
designed 3D printed frames for each gripper. Fig. 4 shows anillustration of the tactile behaviors.
Gentle Grasp
The purpose is grasping an object witha small force. We use the force estimation to stop closing the2

FingerVision for Tactile Behaviors, Manipulation, and Haptic Feedback Teleoperation (Akihiko Yamaguchi)

Fig. 4. Tactile behaviors.

gripper.
Holding
The purpose is controlling the grasp (gripper
width) to avoid slip when holding an object. The strategy issimple: closing the gripper when slip is detected.
Grasp Adaptation
In general, the goal of grasping iseliminating slippage of holding object during lifting up. We
implement this behavior by combining the lifting up motion
and the holding behavior (slip avoidance control) mentioned
above. More concretely, we repeat until the object is lifted
up: moving the object upward slowly; if slippage is detected,
moving the object to the initial height and closing the gripper
slightly. Refer to (17) for more details of grasp adaptation
control.
HandoverThe purpose is naturally passing an object
to a human. FingerVision is used as a trigger to open thegripper. Both force estimation and slip detection are used as
the trigger: if one of them is detected, the gripper is opened.
Combining two modality increases its applicability. Whengrasping an object strongly, force tends to be detected. Whengrasping a light weight object such as an origami crane, slip
tends to be detected.
Tacking Object
The purpose is tracking an object between the fingers. We control the robot arm so that the target
object locates at the center of the fingers. We use the object detection and the pose estimation. The position on thecamera image plane is estimated accurately; we control therobot arm to center the object on the image. For controllingthe height of the object from the camera, we use the area ofthe object on the image. From two FingerVision sensors on
two fingers, we obtain two estimates of object areas on theimages. By controlling the robot to equalize the areas, theobject locates at the center of the fingers.
Tracking Force
The purpose is operating the robot bypushing slightly. We use the force estimate and control therobot towards the pushed direction. We also use the object
detection as a trigger to activate the control, which increases
the safety since the robot does not move when no object is between the fingers. We compared two variations: one uses theforce estimate of Baxter (estimation from joint torque sensors), and the other combines the force estimate of Baxter
and FingerVision. In the latter case, the robot was operated
with smaller force.

In-hand Manipulation
The purpose is changing theorientation of a grasped object without releasing it. Thisis achieved by repeatedly relaxing and tightening the gripper based on the slip estimate until the target orientation isachieved.
More details of the gentle grasp, holding, handover,and in-hand manipulation are described in (16). Its accompanying video is available on https://youtu.be/
L-YbxcyRghQ/ Fig. 5 shows the examples of grasp adaptation control. Its video is available on https://youtu.be/
0sAkec5bpu4 and https://youtu.be/uy32tO9e7O4 The
video of tacking object is available on https://youtu.be/
TAA4YJqEOqg The video of tracking force is available on
https://youtu.be/FQbNV549BQU
4.

Discussion: FingerVision for Teleoperation
with Haptic Feedback

Haptic feedback teleoperation (e.g. (19)) is an important
technology for telesurgery. It should be also useful in otherrobot manipulations. Manipulation tasks in everyday activity are still difficult problems of robotics. A promising
approach to enable robots performing everyday activity islearning from human demonstrations (20) . However transferring human skills is not easy due to the difference of embodiments between humans and robots. Kinesthetic teaching (21)∼(23) makes demonstrations easier but the issues are thelack of haptic feedback and the difficulty of performing a task
by moving the robot body. Teleoperation was used in demonstrating manipulation skills (24) (25) . In (24), it was referred toas teleoperation training where manipulation of cloth wasdemonstrated to a robot by teleoperating with a head mountdisplay. In (25), manipulation skills of dexterous robot hands
were demonstrated to a simulated robot hand by teleoperatingwith Mujoco HAPTIX system (26) , CyberGlove III, and HTC
vive tracker. However haptic feedback was not used in thesework.
Teleoperating a robot without haptic feedback is difficult
especially when there is slippage between the robot hand andthe manipulated object. In the experiments of teleoperating
the Baxter robot with a joy stick to peel a banana (Fig. 6), we
found that the teleoperation is very difficult since the operator
could not sense the state of the grasp. As the consequence,3

FingerVision for Tactile Behaviors, Manipulation, and Haptic Feedback Teleoperation (Akihiko Yamaguchi)
4, 5, pp. 472–484 (1988).
( 3 ) H. Maekawa, K. Tanie, K. Komoriya, M. Kaneko, C. Horiguchi and
T. Sugawara: “Development of a finger-shaped tactile sensor andits evaluation by active touch”, Robotics and Automation, 1992.
Proceedings., 1992 IEEE International Conference on, Vol. 2, pp.
1327–1334 (1992).
( 4 ) K. Nagata, M. Ooki and M. Kakikura: “Feature detection with
an image based compliant tactile sensor”, 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 838–843(1999).
( 5 ) K. Kamiyama, H. Kajimoto, N. Kawakami and S. Tachi: “Evaluation of a vision-based tactile sensor”, Robotics and Automation,
2004. Proceedings. ICRA ’04. 2004 IEEE International Conference
on, Vol. 2, pp. 1542–1547 (2004).
( 6 ) J. Ueda, Y. Ishida, M. Kondo and T. Ogasawara: “Development of
the NAIST-Hand with vision-based tactile fingertip sensor”, Proceedings of the 2005 IEEE International Conference on Robotics
and Automation, pp. 2332–2337 (2005).
( 7 ) C. Chorley, C. Melhuish, T. Pipe and J. Rossiter: “Development ofa tactile sensor based on biologically inspired edge encoding”, Advanced Robotics, 2009. ICAR 2009. International Conference on,
pp. 1–6 (2009).
( 8 ) D. Hristu, N. Ferrier and R. W. Brockett: “The performanceof a deformable-membrane tactile sensor: basic results ongeometrically-defined tasks”, Robotics and Automation, 2000. Proceedings. ICRA ’00. IEEE International Conference on, Vol. 1, pp.
508–513 (2000).
( 9 ) Y. Ito, Y. Kim, C. Nagai and G. Obinata: “Shape sensing by visionbased tactile sensor for dexterous handling of robot hands”, 2010
IEEE International Conference on Automation Science and Engineering, pp. 574–579 (2010).
(10) H. Yussof, J. Wada and M. Ohka: “Sensorization of robotic hand using optical three-axis tactile sensor: Evaluation with grasping andtwisting motions”, Journal of Computer Science, 6, 8, pp. 955–962(2010).
(11) Y. Ito, Y. Kim and G. Obinata: “Robust slippage degree estimation based on reference update of vision-based tactile sensor”, IEEE
Sensors Journal, 11, 9, pp. 2037–2047 (2011).
(12) T. Assaf, C. Roke, J. Rossiter, T. Pipe and C. Melhuish: “Seeing bytouch: Evaluation of a soft biologically-inspired artificial fingertip
in real-time active touch”, Sensors, 14, 2, p. 2561 (2014).
(13) R. Li, R. Platt, W. Yuan, A. ten Pas, N. Roscup, M. A. Srinivasan
and E. Adelson: “Localization and manipulation of small parts using GelSight tactile sensing”, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3988–3993 (2014).
(14) N. F. Lepora and B. Ward-Cherrier: “Superresolution with an optical tactile sensor”, Intelligent Robots and Systems (IROS), 2015
IEEE/RSJ International Conference on, pp. 2686–2691 (2015).
(15) T. Ikai, S. Kamiya and M. Ohka: “Robot control using natural instructions via visual and tactile sensations”, Journal of Computer
Science, 12, 5, pp. 246–254 (2016).
(16) A. Yamaguchi and C. G. Atkeson: “Implementing tactile behaviors
using FingerVision”, the 17th IEEE-RAS International Conferenceon Humanoid Robots (Humanoids’17) (2017). (In press).
(17) A. Yamaguchi and C. G. Atkeson: “Grasp adaptation control with
finger vision: Verification with deformable and fragile objects”,
the 35th Annual Conference of the Robotics Society of Japan(RSJ2017), pp. 1L3–01 (2017).
(18) A. Yamaguchi: “FingerVision”, http://akihikoy.net/p/fv.
html (2017).
(19) A. Bolopion and S. Régnier: “A review of haptic feedback teleoperation systems for micromanipulation and microassembly”, IEEE
Transactions on Automation Science and Engineering, 10, 3, pp.
496–502 (2013).
(20) A. Billard and D. Grollman: “Robot learning by demonstration”,
Scholarpedia, 8, 12, p. 3824 (2013).
(21) S. Calinon, F. Guenter and A. Billard: “On learning the statistical representation of a task and generalizing it to various contexts”,
Proceedings 2006 IEEE International Conference on Robotics andAutomation, 2006. ICRA 2006., pp. 2978–2983 (2006).
(22) M. Mühlig, M. Gienger, S. Hellbach, J. J. Steil and C. Goerick:“Task-level imitation learning using variance-based movement optimization”, the IEEE International Conference on Robotics and Automation (ICRA’09), pp. 1177–1184 (2009).
(23) J. Kober and J. Peters: “Policy search for motor primitives inrobotics”, Machine Learning, 84, 1-2, pp. 171–203 (2011).
(24) P. C. Yang, K. Sasaki, K. Suzuki, K. Kase, S. Sugano and T. Ogata:“Repeatable folding task by humanoid robot worker using deeplearning”, IEEE Robotics and Automation Letters, 2, 2, pp. 397–
403 (2017).
(25) A. Rajeswaran, V. Kumar, A. Gupta, J. Schulman, E. Todorov
and S. Levine: “Learning complex dexterous manipulation withdeep reinforcement learning and demonstrations”, ArXiv e-prints,
arXiv:1709.10087 (2017).
(26) V. Kumar and E. Todorov: “Mujoco haptix: A virtual reality system
for hand manipulation”, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 657–663 (2015).

Fig. 6. Teleoperation of peeling banana with Baxter.
Watch: https://youtu.be/rEeixPBd3hc

the banana slipped often in the grippers which caused many
manipulation failures. This could be improved by installing
tactile sensors on the robot grippers and activating the holding strategy to avoid slippage. However such a control does
not work when the operator wants to produce slippage on
purpose; for example, washing a dish with hand.
Haptic feedback teleoperation for demonstration will increase the variety of teachable tasks and performance of skills(e.g. speed of motions). With a sophisticated haptic feedback, the teleoperators of robots can demonstrate skills as
if moving their own bodies, which is easier than kinesthetic
teaching. Example tasks are peeling banana, mixing stew,
other cooking tasks, washing dishes with hands, and construction of PC (especially handling cables).
4.1 FingerVision for Haptic Feedback
FingerVision
provides multimodal tactile sensation to robot hands. It could
be a useful sensing device to be used in haptic feedback teleoperation.
FingerVision perceives 3-dimensional force distribution
and non-directional slip distribution. Humans can sense thesemodality with their skin, but the sensitivities of FingerVision and human skins are different. FingerVision is sensitive
to slippage regardless the object weight; actually it senses
movement even when the object is not contacting with thefinger. Moreover, FingerVision has other modality such as apose, an area, and texture of nearby object, which human skincannot perceive. Thus, the research question is how to create
haptic feedback from the sensed data of FingerVision? Ideally we will need a haptic device that gives multimodal sensation with high resolution. Another question is that: can humans adapt to use such haptic feedback to teleoperate robots
to manipulate objects better?
It will be also necessary to improve FingerVision for haptic
feedback teleoperation purpose. The current qualities of spatial resolution, dynamic range of force, precision, etc. will
not be enough in some scenarios.
5.

Summary

We described a vision-based tactile sensor FingerVision,and its applications to tactile behaviors and manipulation. We
also discussed the use of FingerVision for teleoperation withhaptic feedback.

References( 1 ) A. Yamaguchi and C. G. Atkeson: “Combining finger vision andoptical tactile sensing: Reducing and handling errors while cutting
vegetables”, the 16th IEEE-RAS International Conference on Humanoid Robots (Humanoids’16) (2016).
( 2 ) S. Begej: “Planar and finger-shaped optical tactile sensors for
robotic applications”, IEEE Journal on Robotics and Automation,

4