May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

International Journal of Humanoid Robotics
c World Scientific Publishing Company

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

Akihiko Yamaguchi
Graduate School of Information Sciences, Tohoku University,
6-6-01 Aramaki Aza Aoba, Aoba-ku, Sendai, Miyagi 980-8579, Japan,
info@akihikoy.net
Christopher G. Atkeson
The Robotics Institute, Carnegie Mellon University,
5000 Forbes Avenue, Pittsburgh PA 15213-3890, United States,
cga@cs.cmu.edu
Received Day Month Year
Revised Day Month YearAccepted Day Month Year
This paper introduces a vision-based tactile sensor FingerVision, and explores its usefulness in tactile behaviors. FingerVision consists of a transparent elastic skin marked
with dots, and a camera, that is easy to fabricate, low cost, and physically robust. Unlike
other vision-based tactile sensors, the complete transparency of the FingerVision skinprovides multimodal sensation. The modalities sensed by FingerVision include distributions of force and slip, and object information such as distance, location, pose, size,
shape, and texture. The slip detection is very sensitive since it is obtained by computer
vision directly applied to the output from the FingerVision camera. It provides highresolution slip detection, which does not depend on the contact force, i.e. it can sense
slip of a lightweight object that generates negligible contact force. The tactile behaviors explored in this paper include manipulations that utilize this feature. For example,
we demonstrate that grasp adaptation with FingerVision can grasp origami, and otherdeformable and fragile objects such as vegetables, fruits, and raw eggs.
Keywords: FingerVision; tactile sensor; tactile behavior.

1. Introduction
We are exploring a vision-based tactile sensor FingerVision1 . Unlike other visionbased tactile sensors such as TacTip2 and GelSight3 , FingerVision has a transparent
skin that enables an embedded camera to see through the skin. This feature increases
the modalities obtained by the sensor, including vision of nearby objects and slip
distribution. We explore if FingerVision is a promising approach to overcome theissues of existing tactile sensors, and how we effectively use FingerVision in designing
robotic behaviors.
FingerVision consists of a transparent elastic material, a transparent hard layer,
and cameras. On the surface of the elastic material, small dots are placed to track
1

May 7, 2019

2

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

the deformation of the material with computer vision. The conceptual diagram isshown in Fig. 1. By processing the video data from the cameras of FingerVision, we
can obtain tactile sensation and vision of nearby objects (proximity vision). The
features of FingerVision can be summarized as follows:(1) Multimodal: It can sense force distribution, high-resolution slip distribution,
object distance, location, pose, size, shape, texture, and other information obtained from proximity vision.
(1.1) Slip can be detected regardless of the force on objects. It can sense slippage even when the object is too light to generate measurable force (e.g.
origami).
(1.2) Cameras can sense objects before collision. With this feature, we cancreate safe interactive robots that are aware of nearby humans and fragile
objects.
(2) Easy to fabricate: Because of its simple structure, its fabrication is easy.
(3) Low cost: The most expensive component is the camera, which is a low costwebcam. Other components are also inexpensive.
(4) Physically strong: External force is applied to the skin and frame, and does
not reach the camera. Thus it is physically strong.
(5) Easy to repair: Even if the skin is damaged, replacing it is inexpensive.
(6) Using wide-angle lenses (fisheye lenses), we can place the cameras sparsely
distributed under the surface where we want to install tactile sensing.
(7) Sensor parameters are adjustable: We can adjust the dynamic range of force(hardness and thickness of the skin), size (small cameras miniaturize the sensorsize), spatial resolution (camera resolution, marker allocation, etc.), and temporal resolution (high speed cameras).
(8) Other types of sensing components can be used, such as range finders andthermal cameras.
(9) Open source: The fabrication process including CAD files of frames and molds,
software, and tutorials are available on-line4 in order to encourage people toreproduce FingerVision for their own robots and projects.
Compared to the other vision-based tactile sensors, the features (1.1), (1.2), (8) are
unique to FingerVision because of its transparent skin.
In this paper, we demonstrate the use of FingerVision, especially tactile manipulation with FingerVision, in order to show its usefulness. The tactile behaviors
presented in this paper are simple to program. Even so, some behaviors are dramatically improved because of the advantages of FingerVision. For example, the grasp
adaptation with FingerVision is very sensitive because of (1.1). FingerVision canadapt a grasp to a range of objects, including lightweight, deformable, and fragile
ones.
Part of this paper were published in conference papers; an early prototype1 ,tactile behaviors5,6 , and grasp adaptation7 . The purpose of this paper is providing
a comprehensive understanding of tactile behaviors with FingerVision.
The rest of this paper is organized as follows: Section 2 reviews the related

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

3

Fig. 1. Conceptual design of FingerVision (a) and an installation sketch on a robotic gripper (b).
Right two images are prototypes of FingerVision installed on a Baxter electric parallel gripper anda Robotiq gripper.

work. Section 3 introduces FingerVision. Section 4 describes the tactile behaviors.
Section 5 reports the results of experiments. Section 6 is a discussion section, andSection 7 concludes the paper.
2. Related Work2.1. Tactile Sensors in General
There are many different approaches of tactile sensing, such as capacitive sensors,piezoresistive sensors, magnetic sensors, piezoelectric sensors, optical and proximity sensors, and vision-based sensors. Some of them are commercialized suchas BarrettHand8 , PR29 , and ReFlex Hand10 . More comprehensive reviews areavailable11,12 . When we design multimodal tactile sensing for robot fingers, weneed to deal with issues of fabrication, wiring, power, size, installation, expense,and physical robustness. We think the vision-based approach is a good approach,since: (1) achieving high resolution (superhuman resolution) is not difficult, (2) the
sensor structure can be simple and fabrication is not difficult, (3) wiring is notproblematic by using well established network infrastructure, (4) buying the partsand fabrication equipment is affordable, (5) the sensing device (camera) is becoming
smaller, cheaper, reliable, and better in resolution and speed, due to the markets for
smart phones and endoscopic surgery, and (6) physically robust since the sensing
device can be isolated from skin deformation.
2.2. Vision-based Tactile Sensors
The idea of using imaging sensors for tactile sensing is decades old. An initialattempt was measuring the frustration of total internal reflection within a waveguideon a sensor surface caused by contact13,14,15,16 . The research trend has shifted tomeasuring displacement of markers placed on the sensor surface with computer
vision, such as using a lattice pattern17 , two-colored dots18 , a single dot19 , andsingle-colored dots20,21,22,23,2,24,25 . Marker displacements are proportional to theexternal force as the displacements are directly caused by the external force. The

May 7, 2019

4

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

resolution of the contact force field is decided by the camera resolution and themarker density. The dynamic range of the force measurement can be controlled bychanging the hardness of the elastic material (softer is more sensitive; cf. 26 ).
Similar to the above work, GelSight was developed by Johnson and Adelson27 . It
consists of a transparent elastomer covered with a opaque skin, which is sensitive tothe surface texture and shape of a contacting object. There has been an application
to robotic manipulation tasks3 , shear force and slip estimation with markers28 , anda slenderized fingertip (GelSlim)29 .
Most of previous vision-based sensors including GelSight occluded the view beyond the sensor itself. This simplifies computer vision since the background of theimage is simplified. It makes sensing robust against external lighting conditions andobject appearance. Some studies used functional membranes. For example, GelSight
used a reflective membrane27 , which was effective to sense the object texture in highresolution. An exception was proposed by Patel and Correll30 where a completelytransparent skin was used. However, it used an array of range finders to measure
the distance to an object and the skin deformation rather than using an imaging
optical device.
In contrast, FingerVision uses cameras to view objects of interest and relies
on computer vision to separate objects from the background. Although it could
be a disadvantage, there are good computer vision functions in publicly available
libraries such as OpenCVa . More importantly, making all of the skin transparent
gives FingerVision another modality, proximity vision. The sensitivity of measuringslip is much improved with this approach as discussed below.
2.3. Slip Detection with Tactile Sensors
Slip detection has been studied for decades. An early approach used a mechanical roller to detect slip31 . An approach using acoustic signals caused by slip was
explored32 . A popular approach is using the vibration caused by slip33,34,35,36,37,38 .
Some vibration approaches used accelerometers34,37 . Approaches to create a mechanism for making slip-detection easier are considered, such as soft skin with atexture34 , soft skin covered with nibs33 , and a flexible link structure36 . In 39,40,28 ,they analyzed an observed force (and torque) to detect slip. Many studies detect
slip by using a distributed sensor array41,42,43 . In 43 , a 44x44 pressure distribution isconverted to an image, and slip is detected by image processing. In 44 , a multi-sensor
fusion approach was proposed where they combined stereo vision, joint-encoders of
the fingers, and fingertip force and torque sensors. In 45 , they developed slip detection using center-of-pressure tactile sensors. In 46 , two BioTac47 sensors are usedand several strategies to detect slip are compared experimentally. BioTac sensors
are also used in 48 , where they developed three types of tactile estimation: finger
forces, slip detection, and slip classification.
a http://opencv.org/

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

5

Similar to ours, 49,22,28 developed methods to detect slip for vision-based tactile
sensors. In 49,22 , slip was estimated from the stick ratio (a ratio of areas of stick andcontact regions). In 22 , the stick ratio was estimated from the displacement of dottedmarkers. The GelSight work28 developed a method to detect slip by thresholding
the entropy of shear (marker) displacement distribution.
In contrast, FingerVision estimates slip by directly analyzing the video fromfingertip cameras. Unlike other vision-based tactile sensors mentioned above, FingerVision does not rely on marker displacement. FingerVision can estimate slip even
if there are no markers on its surface. This feature makes FingerVision special: it
can sense slip of very lightweight objects such as origami whose contact force is too
small to measure.
2.4. Tactile Behaviors
Robotic manipulation with tactile sensing is also studied. A popular task is grasping.
Sometimes grasp execution with tactile sensing is referred to as grasp adaptation.
There are heuristic behavior designs of grasp adaptation50,51 , and a human-inspired
grasp strategy40 which was based on the study of grasp strategy of humans52 . Grasp
adaptation is also called re-grasping53,54 . Grasp adaptation is sometimes designedwith a grasp stability estimator that estimates a quality of grasp from tactile sensor
readings55,56,57 . Typically machine learning approaches are used to construct such
estimators, which requires training samples. Using slip sensation to adapt grasp
is also a popular approach33,49,45,15,58,48,59,60,61 . For example, the slip detection ofan optical tactile sensor was used in grasp adaptation15 where the grasping force
of a robot hand was controlled to avoid slip. An experiment of grasping a paper
cup was conducted, where water was poured into it. It was demonstrated that the
robot adapted the grasp against the increasing weight of water without breaking
the paper cup.
In this paper, we follow a grasp adaptation strategy with slip estimation since
it is simple to implement, and we can emphasize the advantage of slip detection
sensitivity with FingerVision. It is especially remarkable that grasp adaptation with
FingerVision can adapt grasp to a light-weight fragile object (e.g. origami) where
the contact force is too small to measure.
Other manipulation studies with tactile sensors are removing a cap of a bottle15
where slip detection was used, rotating a cylinder with a single-finger robot 62 wherean optical tactile sensor2 was used, in-hand manipulation of a cylinder63 , peg-inhole with slip detection44 , inserting a USB connector into a socket3 where GelSight
was used to estimate the pose of the USB connector in the gripper, and in-handmanipulations43 . In 64 , contour-following control was learned with tactile sensors
and reinforcement learning.
Comparing tactile behaviors with other approaches is difficult since we do nothave the same robot (especially the same robotic hand) and the same tactile sensors.
Constructing a common baseline would be difficult. Therefore in this paper, weMay 7, 2019

6

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

demonstrate tactile behaviors that are enabled with FingerVision.
3. FingerVisionThis section introduces the vision-based tactile sensor FingerVision, its fabrication,and the data processing.
3.1. Overview
FingerVision consists of a transparent elastic material made with silicone, a transparent hard layer made with acrylic, and a camera underneath. Some dots (markers)
are placed on the surface of the elastic material, that are made with plastic beads.
The conceptual diagram is shown in Fig. 1. Unlike other research18,22,2 , we do notplace an opaque material on the surface. The whole skin is transparent except forthe markers, and the cameras can see the external scene through the skin. This paper presents a prototype with an RGB camera, but it can be extended with multiple
cameras and other types of cameras such as thermal and depth cameras.
The markers are captured by the cameras and tracked. This gives us a 3-axis(x,y,z) force measurement at each marker point. By combining multiple marker
measurements, we can estimate torque information. The marker size affects theaccuracy of tracking. In general, a bigger marker is easier to detect. The density ofthe markers determines the resolution of the contact force field. There is a trade-off
between the resolution and the surface transparency. The hardness and the thicknessof the elastic layer affect the marker movement caused by contact force (a softer
layer is more easily deformed by a small force), and determine the dynamic range ofthe contact force measurement. The hard layer is assumed to be fixed on the gripper
so that external force is applied to the elastic and hard layers only and does notaffect the cameras. The physical robustness of the FingerVision sensor is decided
by the elastic and the hard layers. The camera resolution affects the accuracy ofthe marker detection and tracking. The camera frame rate affects the sensing frame
rate. These properties (the marker size and density, the hardness and the thicknessof the elastic and the hard layers, and the camera properties) should reflect the
purpose (task) of each part of the skin. Multiple layers of different materials allow
us to create a “hardening” spring or nonlinear compliance.
3.2. Specification of Prototype
As the elastic material, we use silicone, Silicones Inc. XP-565 that has A-16 Shore
hardness after cure. The effective thickness of the elastic material is 4 mm. The
thickness of the acrylic plate is 2 mm.
For markers, we use black micro plastic beads that are spheres of around 1 mm
diameter. The size varies from 0.5 mm to 1.5 mm. The markers are placed on a 5mm grid.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

7

We use a fisheye lens camera ELP Co. USBFHD01M-L180 that has a USB
interface. It can capture at many different resolutions. We use the mode of 320x240
with MJPG compression.
3.3. Fabrication of FingerVision
Fabrication of the FingerVision sensor consists of the following processes: (1) making
base frames, molds, and an acrylic plate, (2) placing markers on the mold, (3) mixing
silicone resin and pouring it into the mold (note: degassing with vacuum is necessary
to remove air bubbles before pouring), (4) inserting the base frame with the acrylic
plate into the resin in the mold, (5) fixing the base frame on the mold, (6) waiting
for the silicone to cure, (7) removing the silicone and the frame from the mold, andfinally (8) attaching the camera. For the stability of the fabrication and installation
on robotic hands, we design the frames and the molds with a 3D printer. Fig. 2 shows
the 3D printed frames, molds, and after casting the silicone (at the beginning of(6)).
We design a frame to attach the hard layer on the finger of a gripper. The
frame has a place to attach the hard layer made with transparent acrylic, and aconnection structure to the gripper. The latter part depends on the gripper. We
create two versions: one is for the electric parallel gripper of a Baxter robot (astandard gripper), and the other is for the Robotiq 2-finger adaptive robot gripper85 (legacy version). Since we have CAD data for the fingers, we can easily connect
to the frame. The Robotiq gripper has a mount on the fingertip under the original
finger pad. We made a structure to attach the frame to the mount. The frame also
has a mount for a camera. We use a 3D printer (LultzBot Mini, Aleph Objects,Inc.) for producing the frames.
The soft layer is made by casting silicone. We make a mold for casting using a
3D printer to achieve consistent fabrication. However we noticed that the surfacesof 3D printed objects are not smooth enough to make optically clear skin even aftersmoothing with sandpaper. Thus we use a 3D printed mold except for the surfacepart of the soft layer. For the surface part, we use ComposiMold.
In order to increase the durability of the soft layer from peeling, we create
depressions and holes on the sides of the frames so that the silicone locks into them,and we cover the hard layer on both top and bottom with the silicone (see Fig. 2(c)).
Fig. 1 shows our Baxter robot with the FingerVision sensors installed.
3.4. Computer Vision for Processing FingerVision Data
Since the sensing element of FingerVision is a camera, the raw data from FingerVision is an RGB video stream. We use computer vision methods to process
the video. There are two types of computer vision for FingerVision video. One ismarker tracking that estimates the marker displacements from the initial positions.
The displacements of markers are used to estimate the force distribution. The otherMay 7, 2019

8

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

Fig. 2. (a) CAD of frames for a Baxter electric parallel gripper and a Robotiq gripper. (b) Moldfor silicone casting. (c) After pouring silicone into the mold.

is proximity vision that consists of nearby object detection and movement detection. The object detection and tracking provides information about a manipulated
object, such as location, pose, area, and texture. It is also used to distinguish the
movement of a manipulated object and the background.
3.4.1. Marker Tracking
We consider two approaches for marker tracking. One is using the mean shift method
to track marker movement. Initial marker positions are obtained by blob detection.
For each marker, we apply mean shift starting from the previous marker position to
obtain the current marker position. The other approach is applying blob detection
locally for each marker. We consider a small region around the previous marker
position, and apply blob detection to obtain the current marker position.
Both methods are implemented in OpenCV. The mean shift method is available as the cv::meanShift function, and blob detection is available as the
cv::SimpleBlobDetector class. We thought the mean shift approach would bebetter since it is a common tracking method. According to our preliminary test,
marker tracking with mean shift was robust. However it turned out that this approach does not provide good marker position accuracy since cv::meanShift returns an updated object location as integer values. Since the marker movementon the image is small (a few pixels), the movement was jumpy. On the other hand,
cv::SimpleBlobDetector provides the detected blob position as floating-point values. The obtained marker position movement was smooth (see the comparison in
Fig. 3(a)). Thus we chose the blob detection-based approach.
The actual procedure consists of two phases: calibration and tracking. In bothphases, we preprocess the image by rectifying the distortion caused by the fisheyelens, and thresholding to extract black colors as the current markers are black. We
also apply a dilation and an erosion to remove noise.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

(a)

9

(b)

Fig. 3. (a) Comparison of blob tracking based on cv::meanShift (left of each pair) andcv::SimpleBlobDetector (right of each pair). (b) An example of marker movements when a normal
force is applied.

Calibration : The sensor is covered with a white sheet to remove the background.
We apply blob detection method to an entire image. Then we apply the tracking
method to several frames (e.g. 10); if some markers are moving due to environmental noise, they are removed from the marker candidates as they are noisy points.
Typically only a few points are removed. The remaining blobs are considered as
initial markers. Note that during the calibration, we do not move the robot and thewhite sheet is fixed on the sensor surface. If we do not remove these points, they
will be observed as noisy movements, which affects the accuracy of force estimate.
Tracking : Starting from the initial marker positions, we track each marker frameby frame. We consider a small (e.g. 30x30) region of interest (ROI) around theprevious marker position. First we count the non-zero pixels in the ROI and compare
it with the non-zero points of the initial marker. If there is a large difference, we do
not perform marker tracking (i.e. a detection failure). Otherwise we apply the blob
detection method to the ROI. Only one blob is expected; otherwise it is considereda failure. We compare the previous and current blob positions and sizes, and if theirdifference is large, it is considered a failure. Otherwise the blob is considered as the
new marker location.
Post Processing: Force Estimation : From the marker movement, we estimate
an array of forces. The blob detection provides a position and a size of each blob.
The position change is caused by a horizontal (surface) force, while the size change
is caused by a normal force. However, since the size change is subtle compared tothe position change, the normal force estimate based on the size change is noisy andunreliable. An alternative approach approximates the normal force at each marker
with a norm of marker position change. This approximation is useful especially when
taking an average of all the forces. When a normal force is applied to the centerof the skin surface, the markers around the point move radially (Fig. 3(b)). An
average of the horizontal forces in such a case will be close to zero, while an averageof the approximated normal forces will have a useful value. Let dx , dy denote thehorizontal marker movement from the initial position. The force estimate at each

May 7, 2019

10

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

marker is given by:q[fx , fy , fz ] = cx dx , cy d2x + d2y , cz dy(1)where cx , cy , cz denote constant coefficients. Note that fy is the normal force (seeFig. 1 for the coordinate system). We also define an average force and a torqueestimate as:
1 X[fx , fy , fz ]
(2)
f=
N
1 X
r × [fx , fy , fz ]
(3)
τ =
N
where N denotes a number of markers, and r denotes a position of a marker fromthe center of the image.
3.4.2. Proximity Vision
Proximity vision processes an image to obtain information about nearby objects,
such as object colors, textures, shape, position and orientation, movement including
slippage, and deformation. This paper focuses on approximate detection of an object
and its movement. Simple approaches to detecting movement are optical flow andbackground subtraction. Movement detection involves detecting movement of theenvironment and the robot body. For example when moving the robot arm, thecamera in FingerVision will capture background change. Operating the gripper also
causes background change. We need to distinguish the movement of an object frombackground change. We developed a detection and tracking method for an object,
as well as movement detection.
For simplicity, we model an object with a histogram of colors. In most grasping
scenarios, a robot gripper approaches an object, or another agent passes an object
to the gripper. In both cases, the object is seen as a moving object in the cameras ofFingerVision. Thus, we design the object detection and tracking as follows. First wecreate a background model as a histogram of colors. At the beginning of grasping,
we detect moving blobs in the image, compute a histogram of colors of the moving
pixels, and subtract the background histogram. The remaining histogram is added
to the object model. In the tracking phase, we apply the back projection of theobject histogram to the current frame, and thresholding to detect the object. We
describe more details in what follows.
Movement Detection : We found that optical flow and background subtraction
are good at detecting changes in a sequence of images. We compared three implementations based on functions in OpenCV, applying cvCalcOpticalFlowLK to
raw images, cvCalcOpticalFlowLK to edge images detected by the Sobel filter,and cv::BackgroundSubtractorMOG2 to raw images. In many cases, the three approaches provided similar results. In some cases, cv::BackgroundSubtractorMOG2

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

11

Fig. 4. An example of the comparison of three functions.

was slightly better than the others (Fig. 4). We used the background subtraction
approach for movement detection.
Object Model Construction : The object model construction consists of twophases. One is the construction of a background model, which is performed at
the beginning of the experiments. The other is the construction of an object model,
which is performed during each grasping action. Both background and object models
are histograms of colors. We use the hue and saturation components of the HSV color
space to construct the histograms, where the number of bins of hue and saturation
components are 100 and 10 respectively.
The background model is constructed with several adjacent frames (e.g. 3). We
average the histograms of all frames. Let us denote the background histogram modelas Hbg (h, s) where h and s denote hue and saturation bin respectively.
During construction of an object model, the object is assumed to be moving
in the image as we described above. At each frame, we detect the moving points
with the background subtraction method, and calculate the histogram of colors as
Hmv (h, s). We update the object histogram model by:′
Hobj(h, s) = min(255, Hobj (h, s)+fgain max(0, Hmv (h, s) − fbg Hbg (h, s)))(4)′
where Hobj (h, s) and Hobj(h, s) are the current and the updated object histogram
models. At the beginning, Hobj (h, s) is initialized to be zero. The component
max(0, Hmv (h, s) − fbg Hbg (h, s)) computes the remaining histogram after subtracting the background histogram from the color histogram of moving points. The
min(255, . . . ) operation is for normalization. fbg and fgain are constant values, forexample 1.5 and 0.5 respectively.
In order to simplify the timing to start and stop object model construction,
we use an object model made with the latest 200 frames. We stop object model
construction when the robot starts closing the gripper.

Object Tracking : In each frame, we track an object by detecting the pixels similar to the object model. Concretely, we apply a back projection method(cv::calcBackProject) with the histogram of the object Hobj (h, s), and thresholdMay 7, 2019

12

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

Fig. 5. Examples of proximity vision. In each case, the detected object is shown as blue. In pushinga screw driver, a human pushed the object which caused slip. The detected slip is emphasized by
the purple color. We can also see green particles that are pixels detected as moving. They are
considered as background movement since they are outside the detected object region.

the result to remove the uncertain pixels. The remaining pixels are the detected
object. These pixels are used in two ways. One is removing the background change
from the moving points obtained from the background subtraction. For this purpose
we apply an erosion (size 2) and a dilation (size 7) to remove noise and expand theboundary of the object. The other is computing the position and the angle (orientation) of the object. This is done by computing the moment of the object pixels.
Examples of proximity vision are shown in Fig. 5.
4. Tactile Behaviors with FingerVision
We have created several tactile behaviors with FingerVision (cf. Fig. 6). In the following, hBehaviori denotes a behavior.
hGentle Graspi: Grasping an object gently by using force estimation. This is useful when grasping a fragile object.
hHoldingi: Controlling the gripper to avoid slip. This is especially useful whengrasping a deformable and fragile object. It is also effective for grasping light-weightfragile objects.
hGrasp Adaptationi: Automating a lifting-up motion with the slip avoidance control (i.e. hHoldingi). It enables the robot to pick up a range of objects.
hHandoveri: Opening the gripper when a force change or slip is detected. This isuseful when passing an object to humans.
hAutomatic Placingi: Placing an object grasped by the robot.
hAutomatic Cuttingi: Automating a part of cutting motion with tactile sensing.
hIn-hand Manipulationi: Change the orientation of a grasped object by repeat-

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

13

Fig. 6. Conceptual diagrams of tactile behaviors where the modalities used by the behavior andthe brief processes are illustrated.

edly relaxing and tightening the gripper based on the slip estimate.
hTracking Objecti: Centering an object between the robot fingers.
hTracking Forcei: Operating the robot by pushing with a small force.
For simplicity, we use position control on our grippers. In the following behaviors,a small movement of the grippers means a position command to create a minimum
movement.
4.1. hGentle Graspi
The behavior is closing the gripper until one of the FingerVision sensors on thefingers senses a sufficient contact force. FingerVision provides an array of forces(each marker gives 3-dimensional force estimate [fx , fy , fz ]). Rather than using an
average force or torque to detect a small force, detecting a small force on eachmarker is better in this scenario. For robustness against marker tracking noise,
we programmed force tracking as follows: We categorize |fy | (norm of the normal
force) into 4 types: noise level, sufficient contact force, medium force, large force,and give scores 0, 1, 3, 5 respectively. Manually defined thresholds are used in thiscategorization. We defined the condition to stop closing the gripper as that the sumof the scores of the array exceeds a threshold (7 worked well in our experiments).
4.2. hHoldingi
The behavior is that the robot slightly closes the gripper when the FingerVision
sensors detect slippage, otherwise no action is performed by the gripper. For slip
detection we use the number of moving points on the object in the image. If theMay 7, 2019

14

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

number exceeds a threshold, it is recognized as a slip event. This strategy is also
considered as feedback control of slip.
Note that the hHoldingi strategy enables a robot to grasp very light-weightfragile objects such as origami. The idea is that if there is not enough friction
between the object and the fingers, the object will slip when the robot moves thehand. Using the hHoldingi strategy until there is no slip, the robot will be able tomove the object without slip. This approach is applicable even when force estimation
cannot sense the contact force from the object. Thus this could be a strategy to
grasp light-weight fragile objects.
4.3. hGrasp Adaptationi
hGrasp Adaptationi is a control to adapt grasp to an unknown object where thegripper is controlled to avoid slip, i.e. activating the hHoldingi strategy introduced
above. Grasping is considered as a control to prevent slip. With a sensor that can
detect slip, we can create a control strategy to prevent slip by adjusting the grasping
force. As explained in the hHoldingi section, this hGrasp Adaptationi will work witha range of objects, from heavy rigid objects to lightweight, deformable, and fragile
objects.
In the implementation, a lifting-up motion is executed with slip feedback control(hHoldingi). The robot tries to lift up an object with the slip feedback control forthe gripper. If the grasping force is not enough to hold the object, the slip feedback
control adjusts the grasp. We refer to this controller as the hGrasp Adaptationi
controller. Fig. 7 shows the control scheme of hGrasp Adaptationi. First, the robot
tries to bring up the object (BringTest) with the slip feedback control (slipavd).
BringTest is performed slowly so that the gripper can adapt the grasp to theobject. Then the robot lifts it up to the final height (LiftUp). LiftUp is faster thanBringTest, while the slip feedback control is still active to adjust the grasp.
The above behavior is a one-way procedure normally. The implementation(Fig. 7) has an error recovery. During the above two motions, the robot starts
a recovery motion when it detects that the object is considered to be dropped(dropped). The detection of object drop is done by thresholding the ratio of theobject area over its initial value as BringTest starts. If the drop condition is satisfied, the robot opens the gripper (GraspInit), moves the gripper to the initial pose
(ToInit), and closes the gripper to the previous value when the recovery motion
started (GraspPrev). Then the robot restarts from BringTest.
In the state machine Fig. 7, there is another block (Emergency stop), which isactivated when FingerVision detects a large force. Such an event is considered to
be an exception since the robot grasps nothing and the gripper width is the initialvalue which should be greater than the object size. An example scenario of suchan event is when the robot drops the object and it rolls under the finger. Although
the robot finger pushes the object vertically in that case (i.e. only the fingertip ofFingerVision contacts the object), FingerVision can still measure the force. This

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

15

Grasp adaptation
ctrlz(0.02,slow)+ slipavd
reachedBringTest

begin

LiftUp

GraspPrev GraspInit
reached

ctrlz(0,fast)exit(success)droppedreached
ctrlg(gprev)

ctrlz(zfinal,fast)+ slipavd
reached

ctrlg(gopen)

reachedToInitGrasp failure detection
and recovery

exit(failure)large force

Emergency stop

Fig. 7. Control scheme of the hGrasp Adaptationi. The detection of grasp failure and recovery,
and emergency stop are unified.

is possible because the camera of FingerVision has a fisheye lens and the elastic
material propagates the deformation at the fingertip toward the middle part. In
such an event is detected, the state machine is designed to stop immediately.
4.4. hHandoveri
We assume that the gripper already grasps an object, i.e. there are forces applied
to the FingerVision sensors. FingerVision is used as a trigger to open the gripper.
Both force change estimation and slip detection are used as the trigger: if one ofthem is detected, the gripper is opened. Combining two modalities increases its
applicability. When grasping an object strongly, force tends to be detected. Whengrasping a light weight object such as an origami crane, slip tends to be detected.
For force change detection, we compare the force estimate on each marker with
its initial value. We count the number of markers where a difference between those
two values exceeds a threshold. When the number exceeds a threshold (e.g. 5), it
is considered as the trigger. The slip detection is the same as that used in the
hHoldingi behavior.
4.5. hAutomatic Placingi
The purpose of this behavior is placing an object grasped by the robot. Tactile
sensing is useful to detect an event when the object touches with the ground. Suchan event could be estimated with external vision with a model of the object, butthere will be uncertainty in estimating the distance between the object and theground. The approach to use tactile sensing can handle such uncertainty. During

May 7, 2019

16

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

the placing motion (the robot moving the gripper downward), the robot stops the
movement and opens the gripper when contact is detected. The implementation ofthis event detection can be the same as the hHandoveri strategy.

4.6. hAutomatic Cuttingi
We implement the hAutomatic Cuttingi motion of fruits with FingerVision. In thispaper, we simply use tactile sensing to detect a large force applied to the knife.
This detection is useful in two cases. One is detecting the event when the knife
reaches the cutting board. Since the knife is often occluded from the robot vision,
estimating such an event contains uncertainty. The event detection with tactile
sensing can handle such uncertainty. The second is detecting too large force for thegripper to hold the knife. This situation happens when cutting hard materials suchas pumpkins.
We create a cutting controller which (a) starts from a state where the knife heldby the gripper is put above the material, (b) moves the knife downward (cutting
vertically), and then (c) slightly pulls the knife (cutting horizontally). The controller
moves the knife to the initial position in order to repeat the motion several times
when it cannot cut off the material at once. Event detection is used to determine thetransition from (b) to (c). It is implemented as follows: (A) if −(fLx − fLx0 )(fRx −fRx0 ) > 10, or (B) if |τLy | + |τRy | > 4, where fLx and fRx indicate the x-valueof the average force of the left and the right sensors, fLx0 and fRx0 indicate theirinitial values (right before cutting), and τLy and τRy indicate the y-value of theaverage torque of the left and the right sensors. The condition (A) is defined todetect a large force. The threshold is decided from a preliminary experiment. The
condition (B) is introduced to avoid rotational slip of the knife. Fig. 8(a) shows thestate machine of the hAutomatic Cuttingi motion.
When too large force is applied to the knife, it may deform the grasp by moving
the fingers (cf. Fig. 8(b)(a)) or the knife may slip in the fingers (cf. Fig. 8(b)(b)). We
emphasize that FingerVision can be used to detect these situations. For example,
look at the camera view of Fig. 8(b)(b); we can find that the angle of the knife isdifferent from its initial grasp position.

4.7. hIn-hand Manipulationi
We assume that the gripper already grasps an object. The robot repeats the following process until the target angle is achieved. The robot slightly opens the gripper
until if senses a small slip. Since there is a small delay between the gripper motion
and slippage, we insert a short waiting time (0.1 s) after each gripper command.
The method to detect slip is the same as that in the hHoldingi behavior, but thethreshold is halved (i.e. more sensitive). After a short waiting time or when slip is
detected, the robot closes the gripper until slip is not detected.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

17(a) State machine of the hAutomatic Cuttingi motion.

(b) Difficulties in cutting.
Fig. 8. hAutomatic Cuttingi.

4.8. hTracking Objecti
The goal of this behavior is centering an object between the robot fingers. We control
the robot arm to achieve this purpose. We use object detection and pose estimation.
The position in the camera image plane is estimated; we control the robot arm tocenter the object on the image. For controlling the height of the object from thecamera, we use the area of the object on the image. From two FingerVision sensors
on two fingers, we obtain two estimates of object areas on the images. By controlling
the robot to equalize the areas, the object locates at the center of the fingers.
This strategy is a demonstration of proximity vision of FingerVision; the robot
responds to an object that is not in contact with the tactile sensors. Such a function
is only possible with transparent skin. This control will be useful in centering an
object before grasping it. Another application would be inspecting fruits before
picking them.
4.9. hTracking Forcei
The goal of this behavior is operating the robot by pushing with a small force. We
use the force estimate and control the robot to move in the pushed direction. We
also use object detection as a trigger to activate the control, which increases safety

May 7, 2019

18

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

since the robot does not move when no object is between the fingers. We compared
two variations: one uses the force estimate of Baxter (estimation from joint torquesensors), and the other combines the force estimate of Baxter and FingerVision.
In the latter case, the robot was operated with smaller force. This control is ademonstration of using FingerVision in a physical human-robot interaction (HRI)
scenario.
5. Experiments
We conduct some experiments to demonstrate how the proposed tactile behaviors
work with FingerVision. The robots we use here are the Baxter robot of Rethink
Co. and UR3 of Universal Robots. Some of the scenes are shown in Fig. 9 withthe force estimation and the proximity vision views. The videos of experiments areavailable on-line:
https://youtu.be/L-YbxcyRghQ Force estimation, pouring water into a grasped
container, test of proximity vision, hGentle Graspi, hHandoveri, hHoldingi, slipbased grasping, and hIn-hand Manipulationi.
https://youtu.be/uy32tO9e7O4
hGrasp Adaptationi of a flower, an origami
crane, and a hairy rubber toy.
https://youtu.be/0sAkec5bpu4 hGrasp Adaptationi of more than 30 kinds ofobjects.
https://youtu.be/TAA4YJqEOqg Tracking a feather with proximity vision (notouch).
https://youtu.be/FQbNV549BQU
hTracking Forcei behavior (playing Tai Chi
with the robot).
https://youtu.be/V0rwJRv2jdk hAutomatic Placingi (from 0:21), and emergency stop (from 0:43).
https://youtu.be/ifOwQdy9gDg
Early test of FingerVision, and hAutomatic
Cuttingi (from 1:33).
5.1. Robotic System
5.1.1. Sensor Network
The cameras of FingerVision have a USB 2.0 interface. In order to avoid long USB
cables, we place local computers. The local computers send videos obtained fromthe FingerVision cameras to a central computer using Ethernet, and the centralcomputer processes all the videos. In the experiments, we use Raspberry Pi 3Bs as
the local computers, and transmit data through a Gigabit Ethernet network. We use
MJPG-streamerb installed on each Raspberry Pi to capture videos from cameras,
and transmit them using a motion JPEG format. In our test, the final output of
FingerVision data processing (marker tracking and proximity vision) was at 63 FPS
b We use a forked version: https://github.com/akihikoy/mjpg-streamer

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

19

Fig. 9. Scenes of experiments. “Force est” are views of force estimation (red lines show estimatedforces), and “Prox vision” are views of proximity vision (blue regions are detected objects, andpurple points are detected movements).

with 320x240 resolution from four cameras simultaneously. In the experiments, wereduced the FPS to 30 to reduce the computational load.
5.1.2. FingerVision on Baxter
The Baxter robot has two 7 degrees of freedom (DoF) arms. We use its velocity
control mode commanded at 500 Hz. Our Baxter robot has two different grippers.
One is the electric parallel gripper of a Baxter robot (a standard gripper) on theright hand, and the other is the Robotiq 2-finger adaptive robot gripper-85 (legacy
version) on the left hand. Two FingerVision sensors are attached on the fingers
of each gripper. We use two Raspberry Pi 3B computers each of which has two
FingerVision camera connections. Fig. 10(a) shows the Baxter system.
5.1.3. FingerVision on UR3
The UR3 robot has 6 DoF and is driven by joint position or velocity commands.
The robot accepts the joint velocity commands at 125 Hz. A 3D printed gripper
actuated by a Dynamixel servo is mounted on the wrist of the robot that has 1
DoF. The servo is operated using the position control mode at 60 Hz, while theMay 7, 2019

20

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson(a) The Baxter system with four FingerVision sensors.

(b) Universal Robots UR3 with FingerVision.

state is observed at 40 Hz. Two FingerVision sensors are attached on the fingers of
the gripper. These devices are integrated with the control box of UR3, a Raspberry
Pi 3B, and a central computer. Fig. 10(b) shows the UR3 system.
5.2. Evaluating Force Estimation
We evaluate the force estimation using a scale. First we let the robot push the scalevertically to evaluate fy . Second we let the robot hold a stick and push the scalewith it in order to evaluate fz . Similarly we evaluate fx by changing the stick andpushing direction. In each case, we discretely increase the pushing force from around
1 [N] to 20 [N]. We record the force in static situations. For each measurement, we
wait for a few seconds for recording. Fig. 10 shows the results. The values of theweight scale are linearly scaled and offset. We noticed that there was hysteresis.
There were two sources of noise: marker tracking and the robot control.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

21

Fig. 10. Average force trajectories in evaluating fy (left), fz (middle), and fx (right) respectively.
The ∗ mark the scale readings (linearly scaled, and offset). The unit of force is omitted as it is notcalibrated as engineered units.

Fig. 11. Gravity-direction component (z-axis) of the force estimate during the pouring-water experiment. Actual pouring is from 35 to 52 [s]. The unit of force is omitted as it is not calibrated
as engineered units.

5.3. Pouring Water into a Grasped Container
We have the Robotiq gripper of Baxter grasp a container, and then pour water into
the container manually. Fig. 11 shows the gravity-direction component (z-axis) of
the force estimate. Pouring was performed from 35 [s] to 52 [s]. The force gradually
increased. This would be accurate enough to estimate the poured amount of water
during a pouring task.
5.4. Test of Proximity Vision
We explore basic results from proximity vision. We let the Robotiq gripper of Baxter
grasp a screw driver weakly, and move it in the gripper manually. Then we let theMay 7, 2019

22

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

Fig. 12. Force and torque estimate (top) and proximity vision (bottom) during rotating the screw
driver. In the proximity vision graph, there are plots of the object angle (radian) and area obtained
from the moment of object pixels, and the total number of moving pixels (normalized by the imagesize). Object angle obtained by an external camera is also plotted (Mocap-obj-angle). The unitsof force and torque are omitted as they are not calibrated as engineered units.

gripper grasp an empty Coke can, and poke the can 4 times. Fig. 12 shows the resultof rotating the screw driver. We can see that the object angle changes from zero tonegative, to positive, and goes back to zero again. The object angle measured by
an external camera is also plotted in the figure. Around the peaks, the object angle
is different from the estimate by proximity vision. This was because around theseangles, a part of the object was out of the camera view. During rotating the screw
driver, there are positive movement values that are capturing the slippage. The
torque estimate sensed the external torque that rotated the screw driver. Fig. 13
shows the result of poking the Coke can. Since the Coke can was light weight, thehuman poked very weakly. The force and torque estimates did not capture the poke.
However the proximity vision detected the movement as we can see four peaks in
the graph that correspond with the four pokes.
5.5. hGentle Graspi
We test the hGentle Graspi strategy with the Robotiq gripper of Baxter. We havethe robot grasp an empty Coke can, and grasp a paper business card on edge. Bothobjects are soft and will be damaged with even small forces. Fig. 14 shows the force
and torque estimate, and the proximity vision output during hGentle Graspi. The
actual grasp happened at 34.5 [s]. We can see a small change of force around thattime. We can also see movement detection before and during grasping. This wascaused by the approaching motion before the grasp. Similar results are found inMay 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

23

Fig. 13. Force and torque estimate (top) and the proximity vision (bottom) during poking a Cokecan. See the caption of Fig. 12 for the plots.

Fig. 14. Force and torque estimate (top) and the proximity vision (bottom) during gently graspinga Coke can. See the caption of Fig. 12 for the plots.

the card case as shown in Fig. 15. The actual grasping happened at 151 [s]. The
movement detection is less than that of the Coke can case. Since the robot grasped
the card on edge, it appeared only in a small region of the image.

May 7, 2019

24

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

Fig. 15. Force and torque estimate (top) and the proximity vision (bottom) during gently graspinga business card. See the caption of Fig. 12 for the plots.

5.6. hHoldingi Strategy
We demonstrate the hHoldingi strategy by grasping a screw driver. We compare twopatterns: (A) the hGentle Graspi strategy, and (B) the hHoldingi strategy. During
grasping with each pattern, a human pushes the driver several times. The results
are shown in Fig. 16. From the graphs of force and torque estimates, we can see thatstronger external force was applied in (B). The orientation of the object is changing
more in (A). Thus the hHoldingi strategy could reduce slip.
We apply the hHoldingi strategy to grasp a marshmallow where a human pulls
the marshmallow. Fig. 17 shows the result. We can see many slip detections (peaksin Right-movement) from the bottom graph, and the magnitude of grasping force(|fy |) is increasing accordingly in the top graph.
Next we let the robot move a stuffed toy. Moving with the hGentle Graspi strategy, the robot dropped the toy due to a slip. However by activating the hHoldingi
strategy, the robot could hold and move the toy. Fig. 18 shows the force and torqueestimates and the proximity vision output during the motion. We find that there
are several discrete events of slippage, and after each of them, the grasping force(see fy ) was increased. At 545 [s], the robot passed the object to the human. The
area of the object in the image, and the force and torque estimates became zeroafter that.
5.7. Grasping A Fragile Object with the hHoldingi Strategy
We verify our concept that by using the hHoldingi strategy the robot can grasp
a very light-weight fragile object. As such an object, we use an origami crane.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

25

Fig. 16. Results of the hHoldingi strategy. Top two graphs are results of hGentle Graspi, andbottom two graphs are ones of the hHoldingi strategy. In each pair, force and torque estimate(top) and the proximity vision (bottom) are plotted. See the caption of Fig. 12 for the plots.

A human passes an origami crane to the gripper, and the Baxter robot uses thehHoldingi strategy. After grasping it without slip, the robot swings its arm to see
if the hHoldingi strategy is effective. Fig. 19 shows the result. The robot performed
grasping from 142 [s] to 145 [s]. We can see slip detection around 155 [s] and so on,
but the object was kept inside the gripper. From the force and torque estimates,
we cannot see informative changes. This was due to the small weight of the object(1.7 g).
5.8. hGrasp Adaptationi
The previous experiment showed our concept works. Next, we conduct a further
test of the hGrasp Adaptationi controller. We verify that when an adequate grasp

May 7, 2019

26

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

Fig. 17. Force and torque estimate (top) and the proximity vision (bottom) during holding andmoving a marshmallow. See the caption of Fig. 12 for the plots.

Fig. 18. Force and torque estimate (top) and the proximity vision (bottom) during holding andmoving a stuffed toy. See the caption of Fig. 12 for the plots.

pose for an object is given, the hGrasp Adaptationi controller can adapt the grasp
to the object robustly regardless of the object and its properties. We let a humanoperator decide a good grasp pose for a given object with a joystick controller, andthen run the hGrasp Adaptationi controller to pick up the object. We do not tune
the parameters of the controller for each object.
We tested with 30 deformable and fragile objects shown in Fig. 20 including

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

27

Fig. 19. Force estimate (top) and the proximity vision (bottom) during grasping a paper bird. See
the caption of Fig. 12 for the plots.

vegetables, fruits, origami objects, and a raw egg. We use the Robotiq gripper of
Baxter. Initially each object is placed on a table.
We conducted 36 trials: Origami box, Origami crane, Badminton ball, Hairy
rubber toy, Cup cake, Chocolate, Strawberry, Tomato-medium-1, Eggplant@1, Eggplant@2, Zucchini-yellow, Mushroom-1@1, Mushroom-1@2, Egg(raw), Pepper-red1, Oyster mushroom-1, Peach-1, Mushroom-2, Potato-1, Kiwi-1, Tomato-medium-2,
Broccoli@1, Broccoli@2, Oyster mushroom-2, Green pepper-1, Kiwi-2, Pepper-red2, Tomato-big, Banana-1@1, Banana-1@2, Banana-1@3, Green pepper-2@1, Green
pepper-2@2, Peach-2, Potato-2, Banana-2. A label with @N denotes an N-th trialof the same object. Examples of successful grasping are shown in Fig. 20. There
were several failures: (1) Dropped after bringing up: Oyster mushroom-1, Potato-1.
(2) Slippage could not be detected due to a computer vision failure: Eggplant@1(the skin was black), Broccoli@1 and Green pepper-2@1 (the color was similar to
the fingers). (3) Closing gripper did not stop in Banana-1@1 because detecting thedeformation of object as slip. Since the contact force from the table disappeared
when bringing up the banana, the banana skin was deformed slightly. (4) In Banana1@2, dropped during bringing up, and failed to re-grasp since the fingers got stuckat the edge of the object, and the passive joints of the gripper bent. Note that (2)was solved by grasping the green part (Eggplant@2), helping the object detection
manually (Broccoli@2), and just trying again (Green pepper-2@2).
The issues of (2) and (3) will be solved by improving the computer vision method
for: (A) a better object detection and (B) distinguishing slippage and deformation.
The issues of (1) and (4) will be solved by improving the behavior. For example,

May 7, 2019

28

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

Fig. 20. Top: 30 objects used in the experiment. Bottom: Examples of grasp (cup cake, raw egg,oyster mushroom, strawberry, tomato, hairy rubber toy).

testing the grasp stability by shaking the object after grasping will avoid (1). (4)can be solved by optimizing the trajectory of fingertip in re-grasping.

5.9. hHandoveri
We demonstrate the hHandoveri strategy by applying it to a Coke can and a businesscard. Both objects are grasped by the hGentle Graspi strategy, and the card isgrasped on edge. Fig. 21 shows the result. In the Coke can case, the robot started
to open the gripper triggered by the slip detection at 25.1 [s]. In the business cardcase, the opening gripper was triggered by the force change detection at 160 [s]. The
reason could be that the Coke can is slippery, while the slip detection does not work
well with the card when it is grasped on edge. We also investigated other object
cases, and found that when the robot grasped an object strongly, the force-trigger
was often used since the slip rarely happened with such grasps.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

29

Fig. 21. Results of the hHandoveri strategy. Top two graphs show the result of the Coke can case,and bottom tow graphs show the result of the card case. Each of them have force and torqueestimates (top) and the proximity vision (bottom). See the caption of Fig. 12 for the plots.

5.10. hAutomatic Placingi, hTracking Objecti, and hTracking
ForceiWe demonstrate the motions of the hAutomatic Placingi with the UR3. The motion
starts where the robot grasps an origami box. The grasp pose is displaced toward
the fingertip on purpose. In the placing motion, the robot tries to move the gripper
downward until its fingertip is around the table. With this setup, the grasped object
will touch the table before the planned motion ends.
Fig. 22 shows an example of the execution. The left image is the initial pose
where we can see the displaced grasp of the object. At the fourth image, the object
touched the table. Since slip was detected, the robot stopped the placing motion
and opened the gripper. The views of FingerVision at the beginning and at slip

May 7, 2019

30

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

Fig. 22. An execution scene of placing origami box.

detection are shown in the same figure. We can see the points in slip at the frame
when slip was detected.
5.11. hIn-hand Manipulationi
We apply the hIn-hand Manipulationi strategy to rotating a pen. The target angle
is 20 degrees from the current angle. Fig. 23 shows the results of eight runs. In
most cases the achieved angles exceeded the target. This was because the object
movement caused by gravity was fast and the sensing and processing frame rate
was not enough to respond to that, and the gripper response was not fast enough.
5.12. hAutomatic Cuttingi
We conduct an experiment of the hAutomatic Cuttingi with the electric parallel
gripper of Baxter. The target objects are a banana and an apple. Fig. 24 shows ascene of cutting the apple and the corresponding marker tracking result. Fig. 25 andFig. 26 shows the sensor values during cutting a banana and an apple respectively
where the trajectories of the average force (x,z) and torque (y) of the left andthe right FingerVision sensors are shown. Since cutting a banana requires only asmall force, the robot cut it with a single trial, while the robot took four trials in
cutting an apple. In the banana case, the robot stopped moving the knife because
the condition (A) of the hAutomatic Cuttingi motion was satisfied when the knife
hit the cutting board. In the apple case, the condition (A) was mainly satisfied by
the pressure from the cut edge of the apple fresh. In these graphs, the condition

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

31

Fig. 23. Result of hIn-hand Manipulationi. The object orientations obtained by the proximity
vision are plotted per time. The initial orientation is set to be zero.

Fig. 24. Marker tracking result in cutting an apple by the robot. The left image is a view from anexternal camera.

(B) was not satisfied, but it was useful when the initial knife orientation was not
perpendicular.

5.13. hTracking Objecti and hTracking Forcei
We demonstrate the motions of hTracking Objecti and hTracking Forcei. Fig. 27
shows the snapshots of 30 seconds of hTracking Objecti where the object is a pinkfeather performed by Baxter. We can see that the feather is detected by the object
detection algorithm in the FingerVision views. The robot tries to center the object
on image by moving its arm.

May 7, 2019

32

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

Fig. 25. Trajectories of the average force (x,z) and torque (y) of the left and the right FingerVision
sensors during cutting a banana by the robot. There was a single cutting motion around the peakof left x-force.

Fig. 26. Trajectories of the average force (x,z) and torque (y) of the left and the right FingerVision
sensors during cutting an apple by the robot. The robot performed the cutting motion four times
to cut the apple completely. The peaks of left and right x-force correspond with the cutting motion.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

33

Fig. 27. Snapshots of hTracking Objecti.

Fig. 28. Snapshots of hTracking Forcei.

Fig. 28 shows the snapshots of 6 seconds of the hTracking Forcei behavior with
the UR3. The human operator pushed the finger with a small force, and then the
robot moved toward the pushed direction. Although this is a simple demonstration,
it shows the capability of FingerVision in HRI applications.
6. Discussion
6.1. Why are tactile sensors not as popular as vision sensors?
There are popular solutions in robot vision (RGB cameras and OpenCV; depth
cameras and the Point Cloud Library), while there is no de facto standard solutions in tactile sensing for robots. The reasons vary: difficulty to physically install,low durability, poor compatibility among different tactile sensors, unclear usage,
maintenance complexity, programming complexity, and expense. As a result, accumulating the knowledge of tactile sensing for robots is more difficult than that ofrobot vision.

May 7, 2019

34

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

6.2. Improving software improves FingerVision.
This is an interesting feature of vision-based tactile sensors. Typically we need to
improve hardware to improve a sensor, while in vision-based tactile sensors, we can
improve the sensing capability by improving the computer vision software. Of course
high-level sensing capabilities can be improved by software with the other types of
sensors, but vision-based tactile sensors have potential to increase low-level sensing
capabilities. For example, one important part of computer vision for FingerVision
is object detection and tracking. We found that there were some failures in thehGrasp Adaptationi experiments. These failures could be solved by improving thecomputer vision. It also should be mentioned that we can use the recent success ofdeep learning in computer vision with vision-based tactile sensors. Especially with
FingerVision, we can use RGB image processing with FingerVision such as object
recognition, which would be an advantage of FingerVision.
6.3. How does FingerVision improve manipulation?
Through the implementation of tactile behaviors, we found there are some usefulcases of FingerVision in manipulation. Slip detection is useful in many scenarios
to avoid slip, hGrasp Adaptationi, and hIn-hand Manipulationi where the slip iscreated on purpose. It is also used as triggers, for example in hHandoveri and placing behaviors. Force estimation is used as triggers in hHandoveri and hAutomatic
Cuttingi behaviors. It is useful to detect an emergency (collision), which was usedin the hGrasp Adaptationi.
6.4. Should we convert the force estimate to an engineering unit
(e.g. Newtons)?
This depends on the application. We are planning to use machine learning methods
to learn dynamical models (e.g. relation between input gripper motion and output
force changes) for example by using neural networks65 . In this case, obtaining contact force information in engineering units is not necessary. If the force estimates are
consistent enough, we could learn mappings from grasping-action parameters to aforce distribution, and from a force distribution to grasping-quality measurements.
These models would be useful to reason about grasping actions. However using anengineering unit will be generalizable to other situations and other robots, so it isstill beneficial to consider. Accurately estimating force will depend on the contact
location and may require finite element modeling of the sensor structure.
6.5. Accuracy, reliability, and hysteresis
The marker movement in a horizontal direction is easier to track. The vertical force
is more difficult to detect as the changes of the marker visual size are comparably
smaller. A way to improve the vertical force sensitivity is increasing the thicknessof the elastic layer, although it would make the skin heavier.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

35

In the experiments, there were some false detections of the markers. These were
mostly due to the external scene. Increasing the number of markers is helpful for
removing outliers, although it will reduce the transparency accordingly.
We found hysteresis, especially when a strong force was applied. In the cutting
vegetable experiments by the robot, the force estimate changed before and after
cutting a hard material. This would be because the deformation of the soft layer
remained. This was mostly reset after releasing the knife.
Although we demonstrated a basic study of force estimation, we did not identify the range of detectable force and the accuracy of force estimation in dynamic
situations. Also, we did not evaluate the detection of slip. These are left for futurework.
6.6. Can FingerVision grasp objects in the dark or
black/transparent objects?
Installing structured light sources is helpful in dark scenes, although we would needto avoid reflections of light at the surface of the silicone skin and the boundary of
the silicone and acrylic layers.
In case an object is black (the same color as the markers), it will become difficult
to detect the markers. However, using multiple colored markers will make the marker
detection more robust against the object colors. Note that proximity vision stillworks with black objects.
On the other hand, proximity vision will not work with completely transparent
objects. However marker tracking may detect the reaction force from the objects.
Recent progress of computer vision also helps FingerVision to solve these issues.
6.7. Can we distinguish slip and deformation?
Since both slip and deformation appear as movement of pixels in the image, it is
not straightforward to distinguish them. Our current implementation of slip detection detects both as slip. From our experience, we found that the patterns of the
movement of pixels are different between slip and deformation. In the case of slip,
the pixels move in the same direction, while in the case of deformation, the pixels
move inward radially.
6.8. Proximity vision vs. functional membranes
The key aspect of FingerVision is the transparent skin, while other vision-based
tactile sensors use opaque skin. FingerVision also has markers to detect force distribution. A drawback of FingerVision would be the robustness to external light andobject view. As discussed above, we think this drawback can be handled.
Some vision-based tactile sensors use functional membranes such as GelSight27 .
While GelSight provides high-resolution and sensitive texture and shape detection,
FingerVision provides proximity vision and sensitive slip detection. FingerVision

May 7, 2019

36

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

can also combine with other sensors such as depth and thermal cameras. Which is
better? It would be difficult to answer now since it depends on the task.
7. Conclusion
We introduced a vision-based tactile sensor FingerVision and explored tactile behaviors implemented on a Baxter and a Universal Robots UR3 to show its usefulness. FingerVision consists of a transparent elastic material with black dots, anda camera, that is easy to fabricate, low cost, and physically robust. Unlike othervision-based tactile sensors, the complete transparency of the FingerVision skinprovides a multimodal sensation. The modalities sensed by FingerVision include
distributions of force and slip, and object information such as distance, location,pose, size, shape, and texture. The slip detection is very sensitive since it is obtainedby computer vision directly applied to the output of the FingerVision camera. It
provides high-resolution slip detection, which does not depend on the contact force,i.e. it can sense slip of a lightweight object that generates negligible contact force.
The tactile behaviors explored in this paper include manipulations that emphasize
this feature. For example, we demonstrated that grasp adaptation with FingerVision
could grasp origami, and other deformable and fragile objects such as vegetables,
fruits, and raw eggs.
Acknowledgements
Yamaguchi was supported in part by NSK Foundation for the Advancement of
Mechatronics, and JST START grant JPMJST1815. Atkeson was supported byaward IIS-1717066 from the US National Science.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision37

References
1. A. Yamaguchi and C. G. Atkeson, “Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables,” in the 16th IEEE-RAS
International Conference on Humanoid Robots (Humanoids’16), 2016.
2. N. F. Lepora and B. Ward-Cherrier, “Superresolution with an optical tactile sensor,”in Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference
on, 2015, pp. 2686–2691.
3. R. Li, R. Platt, W. Yuan, A. ten Pas, N. Roscup, M. A. Srinivasan, and E. Adelson,“Localization and manipulation of small parts using GelSight tactile sensing,” in 2014
IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014, pp.
3988–3993.
4. A. Yamaguchi, “FingerVision,” http://akihikoy.net/p/fv.html, 2017.
5. A. Yamaguchi and C. G. Atkeson, “Implementing tactile behaviors using FingerVision,” in the 17th IEEE-RAS International Conference on Humanoid Robots (Humanoids’17), 2017.
6. A. Yamaguchi, “FingerVision for tactile behaviors, manipulation, and haptic feedback
teleoperation,” in the 4th IEEJ international workshop on Sensing, Actuation, Motion
Control, and Optimization (SAMCON2018), 2018.
7. A. Yamaguchi and C. G. Atkeson, “Grasp adaptation control with finger vision: Verification with deformable and fragile objects,” in the 35th Annual Conference of the
Robotics Society of Japan (RSJ2017), 2017, pp. 1L3–01.
8. Barrett
Technology,“AboutBarrettHand,”https://www.barrett.com/about-barrethand/, 2018, [Online; accessed Oct-26-2018].
9. Willow Garage, “PR2,” http://www.willowgarage.com/pages/pr2/overview, 2015,[Online; accessed Oct-26-2018].
10. RightHand Robotics, “ReFlex hand,” https://www.labs.righthandrobotics.com/
reflexhand, 2018, [Online; accessed Oct-26-2018].
11. U. Martinez-Hernandez, “Tactile Sensors,” Scholarpedia, vol. 10, no. 4, p. 32398, 2015.
12. Z. Kappassov, J.-A. Corrales, and V. Perdereau, “Tactile sensing in dexterous robot
hands review,” Robotics and Autonomous Systems, vol. 74, pp. 195 – 220, 2015.
13. S. Begej, “Planar and finger-shaped optical tactile sensors for robotic applications,”
IEEE Journal on Robotics and Automation, vol. 4, no. 5, pp. 472–484, 1988.
14. H. Maekawa, K. Tanie, K. Komoriya, M. Kaneko, C. Horiguchi, and T. Sugawara,“Development of a finger-shaped tactile sensor and its evaluation by active touch,”in Robotics and Automation, 1992. Proceedings., 1992 IEEE International Conferenceon, vol. 2, 1992, pp. 1327–1334.
15. H. Yussof, J. Wada, and M. Ohka, “Sensorization of robotic hand using optical threeaxis tactile sensor: Evaluation with grasping and twisting motions,” Journal of Computer Science, vol. 6, no. 8, pp. 955–962, 2010.
16. T. Ikai, S. Kamiya, and M. Ohka, “Robot control using natural instructions via visualand tactile sensations,” Journal of Computer Science, vol. 12, no. 5, pp. 246–254,
2016.
17. K. Nagata, M. Ooki, and M. Kakikura, “Feature detection with an image based compliant tactile sensor,” in 1999 IEEE/RSJ International Conference on Intelligent Robots
and Systems, 1999, pp. 838–843.
18. K. Kamiyama, H. Kajimoto, N. Kawakami, and S. Tachi, “Evaluation of a vision-based
tactile sensor,” in Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004 IEEE
International Conference on, vol. 2, 2004, pp. 1542–1547.
19. J. Ueda, Y. Ishida, M. Kondo, and T. Ogasawara, “Development of the NAISTHand with vision-based tactile fingertip sensor,” in Proceedings of the 2005 IEEE

May 7, 2019

38

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

International Conference on Robotics and Automation, 2005, pp. 2332–2337.
20. C. Chorley, C. Melhuish, T. Pipe, and J. Rossiter, “Development of a tactile sensor
based on biologically inspired edge encoding,” in Advanced Robotics, 2009. ICAR
2009. International Conference on, 2009, pp. 1–6.
21. D. Hristu, N. Ferrier, and R. W. Brockett, “The performance of a deformablemembrane tactile sensor: basic results on geometrically-defined tasks,” in Robotics
and Automation, 2000. Proceedings. ICRA ’00. IEEE International Conference on,vol. 1, 2000, pp. 508–513.
22. Y. Ito, Y. Kim, and G. Obinata, “Robust slippage degree estimation based on reference
update of vision-based tactile sensor,” IEEE Sensors Journal, vol. 11, no. 9, pp. 2037–
2047, 2011.
23. T. Assaf, C. Roke, J. Rossiter, T. Pipe, and C. Melhuish, “Seeing by touch: Evaluationof a soft biologically-inspired artificial fingertip in real-time active touch,” Sensors,
vol. 14, no. 2, p. 2561, 2014.
24. T. Sakuma, F. Von Drigalski, M. Ding, J. Takamatsu, and T. Ogasawara, “A universal
gripper using optical sensing to acquire tactile information and membrane deformation,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), 2018, pp. 1–9.
25. B. W. McInroe, C. L. Chen, K. Y. Goldberg, R. Bajcsy, and R. S. Fearing, “Towards asoft fingertip with integrated sensing and actuation,” in 2018 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), 2018, pp. 6437–6444.
26. OptoForce Co., “White paper: Optical force sensors — introduction to thetechnology,” Tech. Rep., January 2015. [Online]. Available: http://optoforce.com/
27. M. K. Johnson and E. H. Adelson, “Retrographic sensing for the measurement of surface texture and shape,” in 2009 IEEE Conference on Computer Vision and Pattern
Recognition, 2009, pp. 1070–1077.
28. W. Yuan, R. Li, M. A. Srinivasan, and E. H. Adelson, “Measurement of shear and slip
with a GelSight tactile sensor,” in 2015 IEEE International Conference on Robotics
and Automation (ICRA), 2015, pp. 304–311.
29. E. Donlon, S. Dong, M. Liu, J. Li, E. Adelson, and A. Rodriguez, “Gelslim: A highresolution, compact, robust, and calibrated tactile-sensing finger,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
30. R. Patel and N. Correll, “Integrated force and distance sensing using elastomerembedded commodity proximity sensors,” in Proceedings of Robotics: Science andSystems, 2016.
31. M. Ueda, K. Iwata, and H. Shingu, “Tactile sensors for an industrial robot to detect
a slip,” in 2nd International Symposium on Industrial Robots, 1972, pp. 63–76.
32. D. Dornfeld and C. Handy, “Slip detection using acoustic emission signal analysis,” inProceedings. 1987 IEEE International Conference on Robotics and Automation, vol. 4,
1987, pp. 1868–1875.
33. M. R. Tremblay and M. R. Cutkosky, “Estimating friction using incipient slip sensing
during a manipulation task,” in [1993] Proceedings IEEE International Conference on
Robotics and Automation, 1993, pp. 429–434 vol.1.
34. R. D. Howe and M. R. Cutkosky, “Sensing skin acceleration for slip and texture perception,” in Proceedings, 1989 International Conference on Robotics and Automation,
1989, pp. 145–150 vol.1.
35. R. Fernandez, I. Payo, A. S. Vazquez, and J. Becedas, “Micro-vibration-based slip
detection in tactile force sensors,” Sensors, vol. 14, no. 1, pp. 709–730, 2014.
36. ——, Slip Detection in Robotic Hands with Flexible Parts. Cham: Springer International Publishing, 2014, pp. 153–167.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

39

37. A. A. S. Al-Shanoon, S. A. Ahmad, and M. K. b. Hassan, “Slip detection with accelerometer and tactile sensors in a robotic hand model,” IOP Conference Series:
Materials Science and Engineering, vol. 99, no. 1, p. 012001, 2015.
38. R. Fernandez, I. Payo, A. S. Vazquez, and J. Becedas, Slip Detection in a Novel Tactile
Force Sensor. Cham: Springer International Publishing, 2016, pp. 237–252.
39. C. Melchiorri, “Slip detection and control using tactile and force sensors,”
IEEE/ASME Transactions on Mechatronics, vol. 5, no. 3, pp. 235–243, 2000.
40. J. M. Romano, K. Hsiao, G. Niemeyer, S. Chitta, and K. J. Kuchenbecker, “Humaninspired robotic grasp control with tactile sensing,” IEEE Transactions on Robotics,
vol. 27, no. 6, pp. 1067–1079, 2011.
41. E. G. M. Holweg, H. Hoeve, W. Jongkind, L. Marconi, C. Melchiorri, and C. Bonivento,“Slip detection by tactile sensors: algorithms and experimental results,” in Proceedingsof IEEE International Conference on Robotics and Automation, vol. 4, 1996, pp. 3234–
3239.
42. N. Tsujiuchi, T. Koizumi, A. Ito, H. Oshima, Y. Nojiri, Y. Tsuchiya, and S. Kurogi,“Slip detection with distributed-type tactile sensor,” in 2004 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), vol. 1, 2004, pp. 331–336.
43. V. A. Ho, T. Nagatani, A. Noda, and S. Hirai, “What can be inferred from a tactile
arrayed sensor in autonomous in-hand manipulation?” in 2012 IEEE International
Conference on Automation Science and Engineering, 2012, pp. 461–468.
44. T. Hasegawa and K. Honda, “Detection and measurement of fingertip slip in multifingered precision manipulation with rolling contact,” in International Conference on
Multisensor Fusion and Integration for Intelligent Systems (MFI 2001), 2001, pp.
43–48.
45. D. Gunji, Y. Mizoguchi, S. Teshigawara, A. Ming, A. Namiki, M. Ishikawa, , and
M. Shimojo, “Grasping force control of multi-fingered robot hand based on slip detection using tactile sensor,” in 2008 IEEE International Conference on Robotics andAutomation, 2008, pp. 2605–2610.
46. J. Reinecke, A. Dietrich, F. Schmidt, and M. Chalon, “Experimental comparison of slip
detection strategies by tactile sensing with the BioTac on the DLR hand arm system,”in 2014 IEEE International Conference on Robotics and Automation (ICRA), 2014,
pp. 2742–2748.
47. R. Johansson, G. Loeb, N. Wettels, D. Popovic, and V. Santos, “Biomimetic tactile
sensor for control of grip,” Feb 2011, US Patent 7,878,075.
48. Z. Su, K. Hausman, Y. Chebotar, A. Molchanov, G. E. Loeb, G. S. Sukhatme, and
S. Schaal, “Force estimation and slip detection/classification for grip control using
a biomimetic tactile sensor,” in 2015 IEEE-RAS 15th International Conference on
Humanoid Robots (Humanoids), 2015, pp. 297–303.
49. A. Ikeda, Y. Kurita, J. Ueda, Y. Matsumoto, and T. Ogasawara, “Grip force control
for an elastic finger using vision-based incipient slip feedback,” in 2004 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS), vol. 1, 2004, pp.
810–815.
50. P. K. Allen, A. T. Miller, P. Y. Oh, and B. S. Leibowitz, “Integration of vision, force
and tactile sensing for grasping,” Int. J. Intelligent Machines, vol. 4, pp. 129–149,
1999.
51. K. Hsiao, S. Chitta, M. Ciocarlie, and E. G. Jones, “Contact-reactive grasping ofobjects with partial shape information,” in 2010 IEEE/RSJ International Conferenceon Intelligent Robots and Systems, 2010, pp. 1228–1235.
52. R. S. Johansson and J. R. Flanagan, “Coding and use of tactile signals from thefingertips in object manipulation tasks,” Nature Reviews Neuroscience, vol. 10, no.

May 7, 2019

40

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Akihiko Yamaguchi and Christopher G. Atkeson

345, 2009.
53. R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson,
and S. Levine, “More than a feeling: Learning to grasp and regrasp using vision andtouch,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3300–3307, 2018.
54. F. Hogan, M. Bauza Villalonga, O. Canal Anton, E. Donlon, and A. Rodriguez, “Tactile regrasp: Grasp adjustments via simulated tactile transformations,” in IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS), 2018.
55. H. Dang and P. K. Allen, “Stable grasping under pose uncertainty using tactile feedback,” Autonomous Robots, vol. 36, no. 4, pp. 309–330, 2014.
56. M. Li, Y. Bekiroglu, D. Kragic, and A. Billard, “Learning of grasp adaptation throughexperience and tactile sensing,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014, pp. 3339–3346.
57. E. Hyttinen, D. Kragic, and R. Detry, “Estimating tactile data for adaptive graspingof novel objects,” in 2017 IEEE-RAS 17th International Conference on Humanoid
Robotics (Humanoids), 2017, pp. 643–648.
58. C. Chuang, M. Wang, Y. Yu, C. Mu, K. Lu, and C. Lin, “Flexible tactile sensor forthe grasping control of robot fingers,” in 2013 International Conference on Advanced
Robotics and Intelligent Systems, 2013, pp. 141–146.
59. F. Veiga, H. van Hoof, J. Peters, and T. Hermans, “Stabilizing novel objects by learning to predict tactile slip,” in 2015 IEEE/RSJ International Conference on Intelligent
Robots and Systems, 2015, pp. 5065–5072.
60. M. Kaboli, K. Yao, and G. Cheng, “Tactile-based manipulation of deformable objects
with dynamic center of mass,” in 2016 IEEE-RAS 16th International Conference on
Humanoid Robots (Humanoids), 2016, pp. 752–757.
61. J. Venter and A. M. Mazid, “Tactile sensor based intelligent grasping system,” in 2017
IEEE International Conference on Mechatronics (ICM), 2017, pp. 303–308.
62. L. Cramphorn, B. Ward-Cherrier, and N. F. Lepora, “Tactile manipulation withbiomimetic active touch,” in 2016 IEEE International Conference on Robotics andAutomation (ICRA), 2016, pp. 123–129.
63. H. van Hoof, T. Hermans, G. Neumann, and J. Peters, “Learning robot in-hand manipulation with tactile features,” in 2015 IEEE-RAS 15th International Conferenceon Humanoid Robots (Humanoids), 2015, pp. 121–127.
64. R. B. Hellman, C. Tekin, M. van der Schaar, and V. J. Santos, “Functional contourfollowing via haptic perception and reinforcement learning,” IEEE Transactions on
Haptics, vol. 11, no. 1, pp. 61–72, 2018.
65. A. Yamaguchi and C. G. Atkeson, “Neural networks and differential dynamic programming for reinforcement learning problems,” in the IEEE International Conference on
Robotics and Automation (ICRA’16), 2016.

May 7, 2019

22:6

WSPC/INSTRUCTION FILE

ijhr2018a

Tactile Behaviors with The Vision-based Tactile Sensor FingerVision

41

Akihiko Yamaguchi received the BE degree from the Kyoto
University, Kyoto, Japan, in 2006, and the ME and the PhD degrees from Nara Institute of Science and Technology (NAIST),Nara, Japan, in 2008 and 2011, respectively. From April 2010
to July in 2011, he was with NAIST as a JSPS, Japan Society for the Promotion of Science, Research Fellow. From August 2011 to March 2015, he was with NAIST as an AssistantProfessor of the Robotics Laboratory in the Graduate School of Information Science.
From April 2014 to March 2015, he was a visiting scholar of Robotics Institute inCarnegie Mellon University, and from April 2015 to 2017, he was a postdoctoral
fellow of the same institute. From 2017 to present, he is with Tohoku University as
an Assistant Professor of the Graduate School of Information Sciences. His researchinterests include motion learning of robots, reinforcement learning applications torobots, machine learning, artificial intelligence, robotic manipulation, and tactile
sensing.
Christopher G. Atkeson is a Professor in the Robotics Institute and Human-Computer Interaction Institute at Carnegie
Mellon University. He received the M.S. degree in Applied Mathematics (Computer Science) from Harvard University and the Ph.D. degree in Brain and Cognitive Science from M.I.T. He joined the M.I.T. faculty in 1986,
moved to the Georgia Institute of Technology Collegeof Computing in 1994, and moved to CMU in 2000.