June 11, 2019

Advanced Robotics

adr2018a

To appear in Advanced Robotics
Vol. 00, No. 00, Month 2018, 1–17

FULL PAPER
Recent Progress in Tactile Sensing and Sensors for Robotic Manipulation:
Can we turn tactile sensing into vision?
Akihiko Yamaguchia∗ and Christopher G. AtkesonbabGraduate School of Information Sciences, Tohoku University, Miyagi, JapanThe Robotics Institute, Carnegie Mellon University, Pennsylvania, United States(v1.0 released Xxxxzx yyyy)

This paper surveys recently-published literature on tactile sensing in robotic manipulation to understand effective strategies for using tactile sensing and the issues involved in tactile sensing. It consistsof a brief review of existing tactile sensors for robotic grippers and hands, review of modalities available from tactile sensing, review of the applications of tactile sensing in robotic manipulations, anddiscussion of the issues of tactile sensing and an approach to make tactile sensors more useful. We
emphasize vision-based tactile sensing because of its potential to be a good tactile sensor for robots.
Keywords: Tactile sensor, Tactile behavior, Robotic manipulation, Robotic hand

1.

Introduction

In human object manipulation, tactile perception plays an important role in addition to visual
perception [1]. Do robots benefit from tactile sensing? Recent robotic systems are equippedwith good vision systems. Many cameras including RGB, depth, and RGB-D are often usedin robotic application. On the other hand, although there are products of robotic hands wheretactile sensors are embedded, such as BarrettHand [2], PR2 [3], and ReFlex Hand [4], it will be
hard to say that tactile perception and programming robots with tactile sensing are popular.
Why is this?
Although there exists much work on tactile manipulation including machine learning-based
approaches (e.g. [5–8]), the use of tactile perception is not considered as essential in robotics. Or,
robotics engineers seem to be finding ways to avoid using tactile sensors. For example in workon learning robotic grasping with deep learning [9–11], robots learned grasping with visual input
only; tactile sensing was not used. In another study of dexterous in-hand manipulation where a
Shadow Dexterous Hand equipped with tactile (touch) sensors on the fingertips was used, theresearchers did not use the tactile sensors [12]. These implementations were possible because
there were consistent relations between the state before grasping (visual scene of the object andthe gripper), including the manipulation parameters, and the outcome of manipulation. Suchconsistent relations can be learned by neural networks or other machine learning methods.
In the context of learning robotic manipulations from human demonstrations, Yang et al. proposed a method referred to as teleoperation training [13]. Manipulation of clothes was demonstrated to the dual-arm robot (Nextage from Kawada Robotics co.) by teleoperating it witha head mount display. In this research, tactile sensing was not used; tactile sensors were not∗ Corresponding author. Email: info@akihikoy.net

1June 11, 2019

Advanced Robotics

adr2018a

attached on the robots, and the teleoperators did not have tactile feedback. The robots usedvision as the input.
In order to understand effective strategies of using tactile sensors in robotic manipulation,
we survey recently-published research on tactile sensing for robotic manipulations. The focusof this survey is on tactile sensing for robotic manipulation. Rather than surveying on wide
range of tactile sensors, we review off-the-shelf tactile sensors that can be purchased or easily
fabricated. For the review of tactile sensors, refer to surveys such as [14]. This survey emphasizes
an approach of vision-based tactile sensing for robotic manipulation. Taking the recent progressin cameras and computer vision offers the advantages of high resolution, ease of fabrication andwiring, affordability, and reliability.
The rest of this paper consists of: Section 2: We briefly review existing tactile sensors for
robotic grippers and hands. The purpose of this section is knowing what tactile sensors areavailable to non-experts in tactile sensor development. Section 3: We review what modalities areavailable through tactile sensors. We focus on sensing slip. Section 4: We review applications oftactile sensing in robotic manipulation. We focus on grasping with tactile-enabled robotic hands.
Section 5: We discuss two things in this section. One is the issues in introducing tactile sensors torobotic hands. The other is the open source approach of tactile sensor development as a possible
solution to these issues.

2.

Brief Review of Tactile Sensors for Robotic Hands

We briefly review the examples of tactile sensors available in robotic hands and grippers.

2.1

Commercial Tactile Sensor

The dual arm mobile robot PR2 developed by Willow Garage Co. [3] has a parallel jaw gripper
whose fingers have tactile sensors. The tactile sensor on each finger is a capacitive type, consistingof 22 sensing elements. We can obtain a pressure value per sensing element. This tactile sensor was
manufactured by Pressure Profile Systems Inc. A capacitive tactile sensor is an array of sensing
elements each of which is a capacitor made from two metal plates and a dielectric medium.
Applying force on a capacitor changes the distance between the plates, which results in a changeof capacitance. From the measurement of the capacitance change, we can obtain the pressure. It
is considered that implementing a high spatial resolution is difficult with this approach. There
are some applications of capacitive tactile sensors made by Pressure Profile Systems Inc. torobotic hands [15]. An example is the robotic hand with three fingers, BarrettHand developedby Barrett Technology LLC [2]. It optionally has a 6-axis force and torque sensor, fingertip
torque sensors, and tactile sensors. The fingertip torque sensor measures torque of the fingertip
joint with a strain gage. The tactile sensor has 96 sensing elements in total on the fingers and thepalm, that is developed by Pressure Profile Systems Inc. Another example is TWENDY-ONE
of the Sugano Laboratory in Waseda University [16–18]. Each hand has 241 capacitive sensing
elements by Pressure Profile Systems Inc.
The examples of resistive tactile sensors by measuring the change of resistance caused by pressure or deformation are strain gage, force-sensing resistors, pressure conductive elastomer [19],and pressure sensitive ink [20]. There are commercial products of those sensors such as theInastomer from Inaba Rubber Co. [21].
A three-finger robotic hand ReFlex Hand [4] developed by RightHand Robotics Co. provides atactile sensor TakkTile [22]. TakkTile is made with miniature barometric sensor chips consistingof micro electromechanical systems (MEMS) pressure sensors [23]. The sensor is covered withrubber; the pressure sensor senses the pressure applied to the rubber surface. They developed
a unique manufacturing technology: in order to remove the little amount of air in the MEMS
pressure sensor during pouring liquid rubber into the mold, vacuum degassing was used, which2June 11, 2019

Advanced Robotics

adr2018a

was necessary to increase the sensitivity.
Similar to human tactile sensors that have multiple modalities, combining multiple modalities
would be useful in robotic applications. For example BioTac from SynTouch Inc. mimics some ofthe physical properties and sensory capabilities of the human fingertip and gives robotic hands
more human-like tactile sensing [24]. It can measure pressure distribution at 19 points, 3-axis
force, micro vibration, and temperature. SynTouch provides interfaces to commercial robotic
hands such as BarrettHand (Barrett Technology, Inc.) and Shadow Dexterous Hand (Shadow
Robot Company Ltd.). BioTac has a structure in which a core unit consisting of impedance
electrodes, pressure sensor, and thermistor is covered with an artificial skin made of an elastomer;
conductive fluid is filled between the core unit and the skin. A drawback of this sensor is its
expense; it is more expensive than many commercial robotic grippers.
Some researchers use a force and torque sensor mounting on fingertips of robotic hands. Some
small ones are commercially available that are suitable for mounting on fingertips. For example
an optics-based force sensor OptoForce (OnRobot A/S [25]) is mounted on each fingertip of a
Robotiq Three-Finger Adaptive Gripper in [26]. Small force sensors ShokacChip and SchokacCube are distributed by Touchence Inc [27]. These sensors are small and not difficult to installon robotic grippers, however they do not form images of the force or pressure distribution.

2.2

Vision-based Tactile Sensor

When we try to introduce good quality (high-resolution, multi-modal) tactile sensors to robotic
hands, a vision-based approach would be practical since non-experts might be able to manufacture such sensors with adjustments to their robots. Because of the recent progress in imagers,
circuits, and algorithms for computer vision, turning tactile sensing into a vision problem has theadvantages: (1) achieving high resolution (superhuman resolution) is not difficult, (2) the sensor
structure can be simple and manufacturing is not difficult, (3) wiring is not problematic by using
the established network infrastructure, (4) buying the ingredients is affordable, (5) the sensing
device (camera) is becoming smaller, cheaper, reliable, and better in resolution and speed, due tothe markets of smart phone and endoscopic surgery, and (6) physically robust since the sensing
device and the skin can be isolated. Regarding (5), there is an example of embedding a small
camera on a human fingertip [28].
The research of vision-based tactile sensor has decades of history. An initial attempt wasmeasuring the frustration of total internal reflection within a waveguide on a sensor surface
caused by contact [29–32]. The research trend has shifted to measuring displacement of markers
placed on the sensor surface where cameras are used to capture the video and computer vision
is used to detect and track the marker movements [33–42]. A reason for this trend is that it iseasy to acquire a higher spatial resolution estimate of the force field, and measuring the marker
displacements is robust since displacements are proportional to the external force. The resolutionof the contact force field depends on the camera resolution and the marker density. Recently
high resolution sensors have been proposed, such as TacTip [41]. The dynamic range of the force
measurement can be controlled by changing the hardness of the elastic material. For example
using softer elastomer as the as the skin, the sensor becomes more sensitive to smaller force.
GelSight is one of most popular vision-based tactile sensor, which was developed by Johnson
and Adelson in [43]. It consists of a transparent elastomer covered with a reflective skin. The
surface texture and shape of an object can be observed by pressing the object surface on thereflective skin (the observation is made from the elastomer side). It could reconstruct the texturesof a cookie, a decorative pin, a human fingerprint, and a twenty dollar bill (it is not flat!).
GelSight was installed on a finger of a Baxter robot for robotic manipulation tasks [44] where
they used GelSight for localizing an object on the finger. In order to estimate the shear forces
as well as normal and torsional load, markers were placed around the surface (reflective skin) of
the sensor [45]. Similar to other work of vision-based force estimation, they applied computer
vision to detect and track the markers. It was also possible to estimate incipient slip [45]. For

3

June 11, 2019

Advanced Robotics

adr2018a

better robotic manipulation, a version of GelSight with a slenderized fingertip was developed
by introducing mirrors around the fingertip, named GelSlim [46]. The finger was designed to bephysically durable; it could survive more than 3000 grasp trials.
Much of the above work covers a transparent elastic material with an opaque skin in order toremove the effect of the background on the computer vision processes. In contrast, Yamaguchi
and Atkeson proposed the FingerVision sensor which is not covered with an opaque skin [42]. It
means the camera can see through the materials. Markers are placed on the surface. Tracking
the marker movements with computer vision estimates the external force distribution. Additionally, it provides visual information of nearby objects (proximity vision). Analyzing the visual
information gives different modalities, such as high-resolution slip [47], object texture (it could
recognize a QR code on the object), and object pose. It should be emphasized that many otherslip sensing methods are estimated from other modalities (e.g. vibration, pressure distribution),while FingerVision directly measures the movement of an object. Another interesting point isthat even humans do not have such tactile sensation that can see through the skin. FingerVision
is available as open source [48] so that researchers can reproduce the sensor. Models for 3D
printers are designed for Robotiq and Baxter grippers.
Another approach of vision-based tactile sensing was proposed in [49]. They developed a visionbased tactile sensor with a compound-eye camera consisting of an array of lenses and an imager.
Such a camera can sense RGB images and infrared images at the same time. They placed acompound-eye camera under an acrylic plate which was used as a waveguide for infrared light.
Contact on the acrylic plate can be detected based on frustrated total internal reflection [50]from IR images. Since the other RGB imagers see through the acrylic plate, the sensor can sense
nearby objects by stereo vision. This sensor was used in robotic grasp control [51]. The benefit ofthis approach is that it can sense the contact with vision, i.e. it does not depend on the amountof force from the object. Similar to the transparency of the skin of FingerVision, this approach
can be combined with many other ideas such as a vision-based slip detection. A drawback ofthis approach is that it depends on the optical qualities of the object as reported in [50]. We
would need to investigate what objects this approach can sense the contact on. Another difficulty
would be the availability of compound-eye cameras.
Similar to vision-based tactile sensors, there is an approach of using proximity sensors (range
finders and distance sensors). For example, a tactile sensor consisting of range finders covered
with transparent elastic material is proposed in [52]. The idea is that the deformation of thetransparent material can be estimated from the measurement of the distances between therange finders and a nearby object. Normal contact forces are computed from the deformation.
If there is no contact with the object, this sensor simply gives the distances to it. A similar
idea is implemented on a hi-speed robot hand where proximity sensors are used to measure thedistance to any objects, and contact. In [53], tactile sensors with a more dense array of proximity
sensors were developed, which was used to sense the distance to an object and the surface shape.
In [54], fingertips with high-speed and high-precision proximity sensors were developed. They
were installed on a high-speed robotic gripper, which was able to catch a fragile object (a paper
balloon). In [55], they developed sensing fingers with proximity sensors based on their previous
work [56, 57]. Originally the fingertip had a tactile sensor consisting of an array of proximity
sensors (Net-Structured Proximity Sensors [56, 57]) that senses the distance to an object. In
addition, new proximity sensors are added to the fingernail of each finger for measuring the
distance to a support surface such as a table [55].

3.

Modalities of Tactile Sensing

There are various modalities of tactile sensing, including contact (on/off), pressure or force
distribution, slip, vibration, temperature, and object properties such as texture and hardness.
Providing pressure distribution seems to be most common. Some sensors provide a 3-axis force

4June 11, 2019

Advanced Robotics

adr2018a

distribution. Examples of other modalities sensed by tactile sensors are: micro vibration (e.g.
BioTac [24]), temperature (e.g. BioTac [24], TakkTile [22]), and proximity [52].

3.1

Sense of Slip

As well as force distribution, sense of slip is important in object manipulation. Much research
has investigated the role of slip sensation in human manipulation. An important early work on
the role of slip in human grasping is [58] which describes a holding behavior where grip force isincreased when slip is detected. In addition to launching the field, this work eventually led tothe design of the BioTac sensor [59]. Recent papers following up on this work include [60], which
discusses how to hold very light objects, and [61], which discusses how grip force is increased
during arm movements. More information is available in [62, 63].
Many attempts have been made to develop slip sensors. An early approach used a mechanicalroller to detect slip [64]. An approach to use acoustic signals (acoustic emission) caused byslip was explored in [65]. A popular approach is using the vibration caused by slip [66–71].
Some vibration approaches used accelerometers [66, 67, 70]. In [26, 45, 72, 73], they analyzed
an observed force (and torque) to detect slip. For example in [73], high-pass filtered normal
force was used to detect slip. In an optics-based tactile sensor [31], slip is detected as a binary
flag when the change of normal force is greater than a threshold. In [26], slip was detected bymeasuring the increasing rate of change of tangential force.
There are approaches to create a mechanism for making slip-detection easier. In [66], a soft
skin covered with nibs which produce small vibrations was introduced into a fingertip. Two
accelerometers were embedded in the fingertip. A similar structure of nibs was introduced into
an optics-based tactile sensor [31]. In [69], a gripper consisting of two elastic fingers was developed
where strain gauges are embedded at the bottom of fingers. Slip on the fingers is detected by
the strain gauges as vibration.
Detecting slip with a distributed sensor array is also a popular approach [74–77]. In [77], a44x44 pressure distribution is converted to an image, and slip is detected by image processing.
In [78], a multi-sensor fusion approach was proposed where they combined stereo vision, jointencoders of the fingers, and fingertip force and torque sensors. In [76], they developed slip
detection using center-of-pressure tactile sensors. In [79], they trained hidden Markov models to
predict slip. According to their experimental results, the method predicted a slip before a slip
actually took place. Some researchers use the BioTac sensor [59]. In [80], two BioTac sensors wereused and several strategies to detect slip were compared experimentally. BioTac sensors were also
used in [81], where they developed three types of tactile estimation: finger forces, slip detection,and slip classification. They compared machine learning methods to estimate finger forces. They
considered two approaches to detect slip: a force-derivative method and a pressure-vibrationbased method. They used neural networks to classify the slip category, linear or rotational, fromtime-varying BioTac electrode values. A similar slip prediction approach was explored in [82, 83]
where they used random forest classifiers to classify the tactile data from a BioTac into slip andnon-slip classes. They investigated the generalization ability of learned classifier over the objects
including unseen ones.
Slip detection methods have been developed for vision-based tactile sensing. In [84], a method
to detect slip of an object by tracking the dotted markers placed on the object was introduced.
Since they need to place markers on the object, the applications may be limited. In [39, 85, 86],
slip was estimated from the “stick ratio”. The stick ratio is a ratio of areas of stick and contact
regions; the ratio is one when there is no slip, it becomes smaller than one when there is incipient
slip, and when the ratio becomes zero, it is considered as total slip. In [39], the stick ratio wasestimated from the displacement of dotted markers. In the work of GelSight [45], an entropy ofshear (marker) displacement distribution was used to detect slip by thresholding the entropy. It
could distinguish the shear (no slip), incipient/partial slip, and total slip. In [87], a slip detection
method for TacTip was developed where they used the difference of position of each marker

5June 11, 2019

Advanced Robotics

adr2018a

between consecutive frames as the input, and trained a support vector machine to classify thetactile signal into slip or static classes.
The marker displacement was used to detect slip in the above vision-based tactile sensors [39,45, 87]. In contrast, direct analysis of video was used in the work of FingerVision [47]. Since
FingerVision can see through the skin, the movement of object against the sensor is directly
shown in the video. Thus, computer vision methods for analyzing the video, such as optical flow
and background subtraction, are applicable to detect the movement. In order to distinguish the
movement of a foreground object and the background, a color histogram-based object detection
was introduced in [47]. Since this approach does not depend on the marker displacement which
requires a certain amount of load, it was possible to detect slip of very lightweight objects suchas origami arts and dried flowers.

4.

Tactile Sensing in Robotic Manipulation

4.1

Grasping

Grasping is a most basic and popular application of tactile sensing. A grasping behavior may be
decomposed into grasp planning and execution of a planned grasp or grasping behavior. Since
tactile sensing is not available in grasp planning, much of tactile grasp research focuses on theuse of tactile sensing during grasp execution. Using tactile sensing in grasp execution is expected
to reduce uncertainty, such as the pose estimation error of an object [88], unknown content of
a container, and friction coefficient of the object. Using tactile sensing aims to increase thesuccess ratio and the stability of grasp. It is achieved by controlling the grasp parameters suchas a grasp pose, width, and force to hold an object. Grasp execution with tactile sensing issometimes referred to as grasp adaptation or adjustment. There are many attempts. One axis
to understand such attempts is the intervals (control cycle) of executing grasp adjustments fortactile observation. When the interval is short, the adjustment is continuously performed. Suchan adaptation tries to locally find a better grasp around the current grasp. Some criteria areproposed to evaluate a grasp, such as grasp stability and slip amount. When the interval islarger, sometimes it is referred to as re-grasp. After executing a grasp, tactile sensors can beused to assess (evaluate) the result of grasp, which may be used to improve the grasp planning.
4.1.1

Heuristic Grasp Adaptation Strategy

Designing control strategies or policies is an approach of grasp adaptation. In [88], an integrated
manipulation system consisting of a robotic arm and a three-finger robotic hand was presented.
An external vision, finger torque sensors, and tactile sensors on fingers and palm were unified.
They demonstrated the usefulness of the unified system through some experiments, including atask of aligning the lid over the canister and screwing on it. The tactile sensors were used tocompensate the error of estimating poses of objects caused by the uncalibrated stereo vision. In[89], a grasp adaptation was developed with tactile sensors on the PR2 robot.
Similarly, a human-inspired grasp strategy was developed for grasping a range of objects
without breaking them [73]. A study of grasp strategy of humans [1] was considered in their
method. The PR2 robot with the tactile sensors on the fingertips was used. These tactile sensors
provide an array of pressures, while humans use slip sensation in grasping [1]. In [73], they
estimated slip from the pressure array as mentioned in Section 3.1.
In [90], a grasping strategy of 3-finger 8-DoF robotic hand with biomimetic tactile sensors
were developed. They used contact detection on each link of the fingers to create the motion of
fingers to adapt the shape of grasp.
4.1.2

Grasp Adaptation with Slip Detection

Using slip sensation to adapt grasp is a popular approach. A simple control strategy, increasing
the grasping force when slip is detected, works robustly in many situations if the detection of slip

6June 11, 2019

Advanced Robotics

adr2018a

is accurate. Examples include [66, 76, 85, 91, 92]. In much of this work, slip was estimated fromother modalities such as vibration, force, and center of pressure, as mentioned in Section 3.1.
Slip detection for an optics-based tactile sensor was used in grasp adaptation [31] where grasping force of a robot hand was controlled to avoid slip. An experiment of grasping a paper cup was
conducted, where water was poured into the cup. It was demonstrated that the robot adapted
the grasp against the increasing weight of water without breaking the paper cup.
[81] created a behavior to gently pick up objects where they combined finger force estimates,and slip detection and classification. As mentioned in Section 3.1, machine learning was used todetect slip from the tactile reading of the BioTac sensors.
A grasping strategy with slip detection was studied in [26] where multiple-finger grippers
were considered. In the experiments, they used a 3-finger robotic hand (Three-finger Adaptive
Robot Gripper from Robotiq Co.) where OptoForce sensors were attached on the fingertips. This
paper explored grasp adaptation to deformable objects with dynamic centers of mass, such as
containers with liquids.
In [82], they trained a classifier of slip and non-slip classes from the BioTac reading, andapplied the estimation in stabilizing the grip. They considered a situation where a single finger
with BioTac on a 7-DoF robot pushed an object against the wall. The control strategy was rather
simple: when the robot predicts slip, it increases the force in the direction of the contact normal.
They extended this method in [83] where an improved slip classifier was used. They were able
to stabilize a deformable plastic cup where the finger pushed one side and a human pushed theother.
Grasp adaptation with FingerVision was explored in [47, 93]. As mentioned in Section 3.1,
slip was directly detected with FingerVision by analyzing the video from the cameras. As theresult, it could detect slip of very lightweight objects such as origami and flowers. Because of thisfeature, the grasp adaptation with FingerVision could adapt grasp to a range of objects including
lightweight ones. In [47], they demonstrated that the grasp adaptation with FingerVision works
with origami cranes and flowers. In [47], they extended the grasp adaptation control for a wide
variety of objects including deformable and fragile objects such as vegetables including tomatoes,
mushrooms, and zucchinis, fruits, raw eggs, and cupcakes.
4.1.3

Grasp Adaptation with Estimating Friction Coefficient

A grasp adaptation strategy which included estimating the friction coefficient was exploredin [94], which was referred to as a slipping control algorithm. They attempted to estimate thefriction coefficient at the contact points of a grasped object from the observation of a tactile
sensor (an optics-based force and torque sensor was used). Then the estimated friction coefficient
was used in slipping control. The same grasp adaptation strategy was used in [95].
4.1.4Grasp Adaptation with Grasp Stability Estimation

Constructing a grasp stability estimator from tactile reading is also a popular approach ingrasp adaptation. A grasp stability estimator can be used to trigger grasp adaptation control,
as well as to plan a better grasp.
In [96], in order to adapt grasping an object whose pose is observed with uncertainty, agrasp adaptation was developed with tactile feedback of the BarrettHand. An SVM classifier
was learned to estimate grasp stability from tactile data, which was used to determine if theadjustment action is executed. The SVM classifier estimates if the grasp is stable or unstable,
whose input is the grasp features represented by a bag-of-words model of the valid contacts
obtained from the tactile sensing data. The training dataset was constructed with the GraspIt!
simulator where the stability was determined by thresholding the epsilon and the volume qualities
that measures the grasp robustness against disturbances (i.e. less-fragility) and the grasp strength
respectively.
In [97], a grasp adaptation to handle uncertainties of object properties such as the weight andthe contact frictions was proposed. It was designed for a three-finger robotic hand with BioTac
sensors on the fingertips. The grasp adaptation consists of an object-level impedance controller,

7June 11, 2019

Advanced Robotics

adr2018a

which is triggered by a grasp stability estimator. The grasp stability estimator was learned fromsuccessful (stable) grasp example. It is a one-class classifier (Gaussian Mixture Model was used)predicting if the current grasp is stable from an input of grasp features. The grasp featuresconsist of the grasp stiffness and the rest length defined in the virtual frame of an object, andthe tactile reading. This grasp stability estimator was also used in the object-level impedancecontroller where the goal is adapting the grasp stiffness and the rest length so that the grasp
becomes stable.
An adaptive grasping method was proposed in [98] that finds a stable grasp on a novel object.
It consists of two estimation models: a grasp stability estimator from tactile data, and a predictorof tactile data from the current data and an adaptive action. By combining them, an adaptive
action that improves the grasp stability can be found by an optimization.
Grasp stability estimation is also used in triggering re-grasp. In [7], a re-grasping strategy
was studied where the quality of grasp is estimated from the tactile sensing, and the re-grasp is
executed when the grasp is unstable.
Calandra et al. used the GelSight embedded 2-finger parallel gripper to predict grasp success [99]. Later they extended the method for adapting grasp by re-grasping an object [100]. In
both papers, a deep, multimodal convolutional neural network was constructed that predicts theoutcome of a grasp (grasp success probability). The input of the network is multimodal: images
from an external RGB camera and the GelSight cameras. In the re-grasping setting [100], grasp
adjustment actions are also used as input. The neural network was trained with 9000 [99] and6450 [100] grasp trials with the Sawyer robot. The trained neural network was used to predictthe grasp success [99], and select the most promising re-grasp action [100]. A similar re-grasping
strategy was explored in [101] where GelSlim was used.
4.1.5

Complete Grasp Process with Tactile Sensing

A complete process of grasp planning with machine learning and grasp execution with tactile
sensing was explored in [102]. The grasp planning was achieved with deep neural networks that
estimate a stable grasp from an input image. A BarrettHand was used with the Universal Robots
UR5 robot to execute the planned grasp. The tactile sensors of the BarrettHand were used tocorrect the training dataset of the neural networks. The dataset consists of pairs of input andoutput variables, where a label of stable or unstable grasp is necessary as the output. The tactile
sensors were used to assess the grasp. In the grasp assessment, a swinging motion was performed
to test the stability of the grasp. The tactile reading during the motion was used to compute theassessment value. If the assessment value is above a threshold after the swinging motion, thatgrasp is recorded as a stable grasp.
4.2

Other Robotic Manipulations

We briefly explore the use of tactile sensors in robotic manipulations other than grasping. There
are several different uses of tactile sensing, such as: detecting events (touch on finger [31], slip [31],and touch on grasped object [32, 42, 47]), estimating pose and location of touched objects [8,44, 47, 77, 103], using the tactile data in part of the state of reinforcement learning [5, 6, 8], andexploring the workspace without vision [104, 105].
4.2.1

Detecting Events with Tactile Sensors

In [31], tactile sensors were used in opening a cap of a bottle by twisting it, where the tactile
data was used mainly as triggers of actions. For example, events of contact (touch) and slip wereused as triggers.
In [42], a cutting fruits task was implemented where a knife was held by a FingerVision-enabledgripper. The tactile data was used to determine when the knife reaches the cutting board. Suchan event would be difficult to detect with external vision due to occlusion.
In [32, 47], robots with tactile-enabled robotic hands were used in handover tasks which areoften considered in human-robot interaction. In both studies, tactile signals were used as a trigger

8June 11, 2019

Advanced Robotics

adr2018a

to activate the handover motions. In [32], the tactile sensors were used to detect a tapping force
applied to the bottom of the grasped object which triggers the handover motion. In [47], thetactile sensors were used to detect touch events on the grasped object where both force change
and slip were considered as the events. When grasping an object tightly, force change is easier todetect than slip, while when grasping an lightweight object gently, slip is easier to detect thanforce change.
In [90], tactile reading was used to trigger the releasing action of grasped object when placing
it on a table. They also used an event detection in a peg-in-hole task where they detected thehole with tactile reading during sliding the peg (a bottle) on a plane.
4.2.2

Estimating Pose and Location with Tactile Sensors

In [103], as an example manipulation with a TacTip, rolling a cylinder with a single-finger
robot was explored. It achieved a super-resolved manipulation at sub-millimeter accuracy where
the high-resolution tactile sensor was used for localizing the cylinder. Similarly in [47, 77], tactile
sensors were used to estimate the poses of objects in the hand, and the estimate was used inin-hand manipulation of the objects.
In [44], pose estimation of a grasped object was used in an insertion task of a USB connector.
GelSight was used to estimate the pose of the grasped USB connector, and then it was inserted
into a socket.
In [8], a contour-following control was learned with tactile sensors and reinforcement learning.
BioTac was used as the tactile sensor on a robotic hand. It demonstrated some manipulationsof tracing a string and closing a ziplock bag.
Another example of pose estimation of a grasped object from tactile reading was proposed in[106], which was implemented on a 3-finger robotic hand with tactile sensors.
4.2.3

Reinforcement Learning with Tactile Sensors

Since modeling the contact between robot fingers and an object is not easy, machine learning
is sometimes used with tactile sensing. There are examples of using reinforcement learning inorder to learn tactile manipulations.
In [5], reinforcement learning of a robotic scraping task was implemented. The state involved
data from tactile sensors that sense an array of pressures. In [6], in-hand manipulation skills of acylinder were acquired with reinforcement learning where tactile data was used as a part of thestate vector. They used a ReFlex hand that has tactile sensors made from MEMS barometers.
In the work of the contour following of a ziplock bag [8], they converted the tactile reading from
BioTac into discrete states (three spatial relationship between the fingerpad and the zipper),which was used as the state space of reinforcement learning.
Since typically the sensor space of a robotic hand with tactile sensors is high dimensional,
sometimes researchers manually extract features for the state of reinforcement learning. Insteadof a manual feature-set design, an approach to use autoencoders was proposed in [107]. It reduces
the high dimensional sensor space into a low dimensional latent space which is used in the state
space of reinforcement learning. They tested the proposed method in a simple robotic task oftilting a pole where BioTac was used. In [108], they developed an unsupervised learning method
to learn a spatio-temporal representation of a time series of tactile data. In [109], they modeled
the manipulation procedure with POMDP (partially observable Markov decision process) inorder to represent the latent states, and built a Q-learning to learn a policy with the learned
representation. In [110], they built a three-dimensional latent space from a tactile sensor where
BioTac was used. Then they learned dynamical models on the latent space for tactile servoing.
4.2.4

Exploring the Workspace without Vision

There is some work to explore a work space in a dark scene, i.e. without vision. In [104], active
exploration was developed to search for objects in an unknown workspace with tactile sensing.
A similar approach was used in a study of tactile-based grasp learning where touch based object
localization called “touch-scans” and tactile based re-grasping methods were developed [105].

9June 11, 2019

Advanced Robotics

4.3

adr2018a

Tactile Perception

We briefly review the applications of tactile sensing to perceiving objects and environment. Fora thorough review of tactile perception, refer to [111, 112]. The basic idea is pressing the tactile
sensors on an object or stroking the surface of an object with the tactile sensors, and estimating
the properties such as texture type, hardness, and shape from the tactile data.
In [38], they explored estimating the local shape (e.g. an edge) of an object and the irregularityof the object surface by pressing a vision-based tactile sensor on the object. In [40], a similar
approach was explored, where the object shapes were detected in more detail. They used TacTip,and referred to their method as “seeing by touch”.
Sensing by stroking or poking a target object is a kind of active perception. In [113], a method
to estimate the classes of wood, cork, paper, and vinyl was proposed where the tactile data wasobserved by stroking the material with a tactile sensor consisting of distributed strain gauges
and PVDF films. In [114], the shape of an object was estimated by stroking the surface withrobotic fingers with tactile sensors. They used optical tactile sensors. In [115], they used theall modalities of a BioTac sensor and estimated 117 kinds of textures at the precision of 95.4%
from tactile data of stroking a texture. In [116], a fingertip with two force sensors, an actively
heated temperature sensor, and a microphone was used to categorize objects into six material
classes; metal, plastic, wood, glass, ceramic, and fabric. In [117], tactile sensors on a four-finger
robotic gripper were used to classify the object into 20 classes. They used machine learning
to classify where several different setups were compared, such as using single frame data andtime-series data. The tactile sensors are called uSkin, which can sense a distribution of 3-axis
force by detecting the magnetic field changes [118].
In [119], they classified grasped objects from tactile reading, which was handled as images.
The tactile sensor was Weiss Robotics sensor DSA 9205, a resistive pressure array consisting of
6 x 14 cells.
As mentioned before, GelSight is able to detect the texture and the shape of an object surface.
It was able to detect the textures of a cookie, a decorative pin, a human fingerprint, a twenty
dollar bill, USB connector, and so on [43, 44]. In [120],they developed a method to estimate
the properties of clothing including thickness, smoothness, fuzziness, season to be used, textile
type, and washing method from the reading of GelSight by pressing it on a cloth. Deep neuralnetworks were used as the classifier where the GelSight image was directly used as the input.
Estimating the shapes of objects by touch (and stroking) is also a popular application of tactile
sensing. As mentioned above, in [40, 43, 44, 114], they used tactile sensors to sense the surface
shape of objects. In [40, 43, 44], they estimated small-scale surface shapes (textures), while in[114], they estimated relatively larger shapes.
In [121], a method to estimate 3D shape of objects was developed where GelSight was usedwith an external vision. In their system, a rough 3D shape of object is predicted from a singleview color image with neural networks, and then it is refined by touching regions where thevisual prediction is highly uncertain.
In [122], a planning algorithm was proposed to estimate a path to stroke an object in order toreduce the uncertainty of the shape roughly modeled from vision. A similar approach of such atactile exploration was proposed in [123].

5.

Discussion

5.1

Issues of Introducing Tactile Sensors to Robotic Hands

Summarizing the survey described so far, we found that there is much work of tactile sensing
and many different uses in the applications to robotic manipulations. However, tactile sensing
seems to be still experimental in robotics. The following are possible reasons:• Difficulty to install on robotic hands. When installing tactile sensors on robotic hands,

10June 11, 2019

Advanced Robotics

adr2018a

we need to place tactile sensors in a limited space, and the finger surface shapes vary including
flat and curved surfaces. Robot hands are typically not designed for sensors.
• Wiring, power supply, and processing. Installing many tactile sensors increases thecomplexity of wiring of signals, power supply, and processing circuits.
• Low durability, fragility. Many of tactile sensors directly interact with external forces,
which sometimes breaks the sensors. If we cover a sensor with elastic material to protect itfrom large external forces, it will decrease the sensitivity and increase the hysteresis. Notethat there are different types of durability, such as external force, liquids, temperature, dirt,
and chemical. If repairing the sensor is low cost and easy, low durability might be accepted.
However many of tactile sensors are not easy to repair. Low durability would also suffer
the use in machine learning due to the change of sensor properties. When a sensor propertychanges, that change needs to be estimated.
• Less compatibility with the other tactile sensors. There are variations of modalities,
and spatial and temporal resolutions in tactile sensors. The sensing principles also vary.
Due to that, many tactile sensors are not compatible with others. They also decrease thereusability of software.
• It is unclear what we can do with tactile sensors. Many robotic manipulations canbe implemented without tactile sensors. Grasp adaptation can be also achieved with soft
robotic hands without tactile sensors.
• Disadvantages caused by tactile sensors.
◦ Maintenance becomes complicated. We would need periodic calibrations, and repairs.
◦ Programming becomes complicated.
• Expensive. High performance tactile sensors are typically expensive.
• Asking for the moon. Some of researchers tend to require tactile sensing capabilities
that are hard to achieve. Implementing human-quality tactile sensors is impossible with thestate-of-the-art.
These issues prevent the researchers from using tactile sensors, which makes it difficult toaccumulate the knowledge of using tactile sensors.

5.2

Open Source Tactile Sensor Project

A solution to popularize the use of tactile sensing in robotic manipulation scenarios would be
open source projects for tactile sensors. Since there are a lot of robotic grippers and hands, providing tactile sensors for all of them is impractical. In many cases, roboticists wish to customize
grippers and hands for their purposes. Making the fabrication, software, and case studies open
would be a solution to this issue, which might lead to forming a community for that tactile
sensor.
Some tactile projects are available from the Soft Robotics Toolkit [124], which is an open
platform of soft robotics. For example TacTip and TakkTile are available through the Toolkit(note that TakkTile is currently a product [22]).
FingerVision is also available as open source [48]. On the project website, its fabrication including CAD models for 3D printing, software including standalone programs and ROS packages,
and tutorials are available. There is a community of FingerVision developers where technicalknowledge about FingerVision development and applications are shared. Some successors ofFingerVision have been developed such as [125].

6.

Conclusion

In this paper, we investigated tactile sensors for robotic hands and their applications to robotic
manipulations. We found that many different types of tactile sensors were used in robotic ma-

11

June 11, 2019

Advanced Robotics

adr2018anipulations in many different ways. Through this survey, we discussed the issues of introducing
tactile sensors into robotic systems. As a possible solution to these issues, an approach of open
source tactile sensors was introduced.

Acknowledgement
Yamaguchi was supported in part by NSK Foundation for the Advancement of Mechatronics,
and JST Program for Creating STart-ups from Advanced Research and Technology. Atkeson
was supported by award IIS-1717066 from the US National Science Foundation.

References[1] Johansson RS, Flanagan JR. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience. 2009;10(345).
[2] Barrett Technology. About BarrettHand. https://www.barrett.com/about-barrethand/. 2018.
[Online; accessed Oct-26-2018].
[3] Willow Garage. PR2. http://www.willowgarage.com/pages/pr2/overview. 2015. [Online; accessed Oct-26-2018].
[4] RightHand Robotics. ReFlex hand. https://www.labs.righthandrobotics.com/reflexhand.
2018. [Online; accessed Oct-26-2018].
[5] Chebotar Y, Kroemer O, Peters J. Learning robot tactile sensing for object manipulation. In: 2014
ieee/rsj international conference on intelligent robots and systems. 2014.
[6] van Hoof H, Hermans T, Neumann G, Peters J. Learning robot in-hand manipulation with tactile
features. In: 2015 IEEE-RAS 15th international conference on humanoid robots (humanoids). 2015.
p. 121–127.
[7] Chebotar Y, Hausman K, Kroemer O, Sukhatme G, Schaal S. Regrasping using tactile perception
and supervised policy learning. In: Aaai spring symposium series. 2017.
[8] Hellman RB, Tekin C, van der Schaar M, Santos VJ. Functional contour-following via haptic perception and reinforcement learning. IEEE Transactions on Haptics. 2018;11(1):61–72.
[9] Lenz I, Lee H, Saxena A. Deep learning for detecting robotic grasps. The International Journal of
Robotics Research. 2015;34(4-5):705–724.
[10] Pinto LJ, Gupta A. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot
hours. In: the IEEE international conference on robotics and automation (icra’16). 2016.
[11] Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D. Learning hand-eye coordination for robotic
grasping with deep learning and large-scale data collection. The International Journal of Robotics
Research. 2017;:0278364917710318.
[12] Andrychowicz M, Baker B, Chociej M, Józefowicz R, McGrew B, Pachocki JW, Petron A, Plappert
M, Powell G, Ray A, Schneider J, Sidor S, Tobin J, Welinder P, Weng L, Zaremba W. Learning
dexterous in-hand manipulation. ArXiv e-prints. 2018;(arXiv:1808.00177). 1808.00177.
[13] Yang PC, Sasaki K, Suzuki K, Kase K, Sugano S, Ogata T. Repeatable folding task by humanoid
robot worker using deep learning. IEEE Robotics and Automation Letters. 2017;2(2):397–403.
[14] Dahiya RS, Metta G, Valle M, Sandini G. Tactile sensing—from humans to humanoids. IEEE
Transactions on Robotics. 2010;26(1):1–20.
[15] Pressure Profile Systems Inc. Tactile sensor applications. https://pressureprofile.com/
applications. 2018. [Online; accessed Oct-26-2018].
[16] Iwata H, Sugano S. Design of human symbiotic robot twendy-one. In: 2009 IEEE international
conference on robotics and automation. 2009. p. 580–586.
[17] Kojima K, Sato T, Schmitz A, Arie H, Iwata H, Sugano S. Sensor prediction and grasp stability
evaluation for in-hand manipulation. In: 2013 IEEE/RSJ international conference on intelligent
robots and systems. 2013. p. 2479–2484.
[18] Sugano Laboratory, WASEDA University. TWENDY-ONE. http://www.twendyone.com/. 2007.
[Online; accessed Oct-26-2018].
[19] Shimojo M, Namiki A, Ishikawa M, Makino R, Mabuchi K. A tactile sensor sheet using pressure
conductive rubber with electrical-wires stitched method. IEEE Sensors Journal. 2004;4(5):589–596.

12

June 11, 2019

Advanced Robotics

adr2018a[20] Gräbner D, Tintelott M, Dumstorff G, Lang W. Low-cost thin and flexible screen-printed pressure
sensor. In: Eurosensors 2017. Vol. 1. 2017.
[21] INABA RUBBER Co Ltd. INASTOMER (pressure conductive sensor). http://www.
inaba-rubber.co.jp/en/b_products/inastomer/. 2010. [Online; accessed Oct-26-2018].
[22] RightHand
Labs.
Takktile
sensors.
https://www.labs.righthandrobotics.com/
takktile-sensors. 2018. [Online; accessed Oct-26-2018].
[23] Tenzer Y, Jentoft LP, Howe RD. The feel of mems barometers: Inexpensive and easily customized
tactile array sensors. IEEE Robotics Automation Magazine. 2014;21(3):89–95.
[24] SynTouch INC. Sensor technology. https://www.syntouchinc.com/sensor-technology/. 2018.
[Online; accessed Oct-26-2018].
[25] OnRobot. https://onrobot.com/. 2018. [Online; accessed Oct-26-2018].
[26] Kaboli M, Yao K, Cheng G. Tactile-based manipulation of deformable objects with dynamic centerof mass. In: 2016 IEEE-RAS 16th international conference on humanoid robots (humanoids). 2016.
p. 752–757.
[27] Touchence Inc. http://www.touchence.jp/en/. 2018. [Online; accessed Oct-26-2018].
[28] Yang XD, Grossman T, Wigdor D, Fitzmaurice G. Magic finger: Always-available input throughfinger instrumentation. In: Proceedings of the 25th annual acm symposium on user interface softwareand technology. 2012. p. 147–156.
[29] Begej S. Planar and finger-shaped optical tactile sensors for robotic applications. IEEE Journal on
Robotics and Automation. 1988;4(5):472–484.
[30] Maekawa H, Tanie K, Komoriya K, Kaneko M, Horiguchi C, Sugawara T. Development of a fingershaped tactile sensor and its evaluation by active touch. In: Robotics and automation, 1992. proceedings., 1992 IEEE international conference on. Vol. 2. 1992. p. 1327–1334.
[31] Yussof H, Wada J, Ohka M. Sensorization of robotic hand using optical three-axis tactile sensor:
Evaluation with grasping and twisting motions. Journal of Computer Science. 2010;6(8):955–962.
[32] Ikai T, Kamiya S, Ohka M. Robot control using natural instructions via visual and tactile sensations.
Journal of Computer Science. 2016;12(5):246–254.
[33] Nagata K, Ooki M, Kakikura M. Feature detection with an image based compliant tactile sensor.
In: 1999 IEEE/RSJ international conference on intelligent robots and systems. 1999. p. 838–843.
[34] Kamiyama K, Kajimoto H, Kawakami N, Tachi S. Evaluation of a vision-based tactile sensor. In:
Robotics and automation, 2004. proceedings. icra ’04. 2004 IEEE international conference on. Vol. 2.
2004. p. 1542–1547.
[35] Ueda J, Ishida Y, Kondo M, Ogasawara T. Development of the NAIST-Hand with vision-based
tactile fingertip sensor. In: Proceedings of the 2005 IEEE international conference on robotics andautomation. 2005. p. 2332–2337.
[36] Chorley C, Melhuish C, Pipe T, Rossiter J. Development of a tactile sensor based on biologicallyinspired edge encoding. In: Advanced robotics, 2009. icar 2009. international conference on. 2009.
p. 1–6.
[37] Hristu D, Ferrier N, Brockett RW. The performance of a deformable-membrane tactile sensor: basic
results on geometrically-defined tasks. In: Robotics and automation, 2000. proceedings. icra ’00.
IEEE international conference on. Vol. 1. 2000. p. 508–513.
[38] Ito Y, Kim Y, Nagai C, Obinata G. Shape sensing by vision-based tactile sensor for dexterous handling of robot hands. In: 2010 IEEE international conference on automation science and engineering.
2010. p. 574–579.
[39] Ito Y, Kim Y, Obinata G. Robust slippage degree estimation based on reference update of visionbased tactile sensor. IEEE Sensors Journal. 2011;11(9):2037–2047.
[40] Assaf T, Roke C, Rossiter J, Pipe T, Melhuish C. Seeing by touch: Evaluation of a soft biologicallyinspired artificial fingertip in real-time active touch. Sensors. 2014;14(2):2561.
[41] Lepora NF, Ward-Cherrier B. Superresolution with an optical tactile sensor. In: Intelligent robots
and systems (iros), 2015 IEEE/RSJ international conference on. 2015. p. 2686–2691.
[42] Yamaguchi A, Atkeson CG. Combining finger vision and optical tactile sensing: Reducing andhandling errors while cutting vegetables. In: the 16th ieee-ras international conference on humanoid
robots (humanoids’16). 2016.
[43] Johnson MK, Adelson EH. Retrographic sensing for the measurement of surface texture and shape.
In: 2009 IEEE conference on computer vision and pattern recognition. 2009. p. 1070–1077.
[44] Li R, Platt R, Yuan W, ten Pas A, Roscup N, Srinivasan MA, Adelson E. Localization and manipulation of small parts using GelSight tactile sensing. In: 2014 IEEE/RSJ international conference

13June 11, 2019

Advanced Robotics

adr2018aon intelligent robots and systems. 2014. p. 3988–3993.
[45] Yuan W, Li R, Srinivasan MA, Adelson EH. Measurement of shear and slip with a GelSight tactile
sensor. In: 2015 IEEE international conference on robotics and automation (icra). 2015. p. 304–311.
[46] Donlon E, Dong S, Liu M, Li J, Adelson E, Rodriguez A. Gelslim: A high-resolution, compact,robust, and calibrated tactile-sensing finger. In: IEEE/RSJ international conference on intelligent
robots and systems (iros). 2018.
[47] Yamaguchi A, Atkeson CG. Implementing tactile behaviors using FingerVision. In: the 17th ieee-ras
international conference on humanoid robots (humanoids’17). 2017.
[48] Yamaguchi A. FingerVision. http://akihikoy.net/p/fv.html. 2017.
[49] Shimonomura K, Nakashima H. A combined tactile and proximity sensing employing a compoundeye camera. In: Sensors, 2013 ieee. 2013.
[50] Han JY. Low-cost multi-touch sensing through frustrated total internal reflection. In: Proceedingsof the 18th annual acm symposium on user interface software and technology. 2005. p. 115–118.
[51] Shimonomura K, Nakashima H, Nozu K. Robotic grasp control with high-resolution combined
tactile and proximity sensing. In: 2016 ieee international conference on robotics and automation(icra). 2016.
[52] Patel R, Correll N. Integrated force and distance sensing using elastomer-embedded commodity
proximity sensors. In: Proceedings of robotics: Science and systems. 2016.
[53] Hirai Y, Suzuki Y, Tsuji T, Watanabe T. High-speed and intelligent pre-grasp motion by a robotic
hand equipped with hierarchical proximity sensors. In: 2018 ieee/rsj international conference on
intelligent robots and systems (iros). 2018.
[54] Koyama K, Shimojo M, Senoo T, Ishikawa M. High-speed high-precision proximity sensor for detection of tilt, distance, and contact. IEEE Robotics and Automation Letters. 2018;3(4):3224–3231.
[55] Sasaki K, Koyama K, Ming A, Shimojo M, Plateaux R, Choley J. Robotic grasping using proximity sensors for detecting both target object and support surface. In: 2018 ieee/rsj international
conference on intelligent robots and systems (iros). 2018.
[56] Hasegawa H, Suzuki Y, Ming A, Koyama K, Ishikawa M, Shimojo M. Net-structure proximity
sensor: High-speed and free-form sensor with analog computing circuit. IEEE/ASME Transactions
on Mechatronics. 2015;20(6):3232–3241.
[57] Koyama K, Suzuki Y, Ming A, Shimojo M. Integrated control of a multi-fingered hand and arm
using proximity sensors on the fingertips. In: 2016 ieee international conference on robotics andautomation (icra). 2016.
[58] Westling G, Johansson RS. Factors influencing the force control during precision grip. Experimental
Brain Research. 1984;53(2):277–284.
[59] Johansson R, Loeb G, Wettels N, Popovic D, Santos V. Biomimetic tactile sensor for control ofgrip. 2011 Feb. US Patent 7,878,075.
[60] Hiramatsu Y, Kimura D, Kadota K, Ito T, Kinoshita H. Control of precision grip force in liftingand holding of low-mass objects. PLOS ONE. 2015;10:1–19.
[61] Slota GP, Latash ML, Zatsiorsky VM. Grip forces during object manipulation: experiment, mathematical model, and validation. Experimental Brain Research. 2011;213(1):125–139.
[62] Chinellato E, del Pobil AP. The visual neuroscience of robotic grasping: Achieving sensorimotor
skills through dorsal-ventral stream integration. [Ph.D. thesis]. Springer. 2008.
[63] Nowak DA. Sensorimotor control of grasping: physiology and pathophysiology. Cambridge University Press. 2009.
[64] Ueda M, Iwata K, Shingu H. Tactile sensors for an industrial robot to detect a slip. In: 2nd international symposium on industrial robots. 1972. p. 63–76.
[65] Dornfeld D, Handy C. Slip detection using acoustic emission signal analysis. In: Proceedings. 1987
IEEE international conference on robotics and automation. Vol. 4. 1987. p. 1868–1875.
[66] Tremblay MR, Cutkosky MR. Estimating friction using incipient slip sensing during a manipulation
task. In: [1993] proceedings IEEE international conference on robotics and automation. 1993. p.
429–434 vol.1.
[67] Howe RD, Cutkosky MR. Sensing skin acceleration for slip and texture perception. In: Proceedings,
1989 international conference on robotics and automation. 1989. p. 145–150 vol.1.
[68] Fernandez R, Payo I, Vazquez AS, Becedas J. Micro-vibration-based slip detection in tactile force
sensors. Sensors. 2014;14(1):709–730.
[69] Fernandez R, Payo I, Vazquez AS, Becedas J. Slip detection in robotic hands with flexible parts.
Cham: Springer International Publishing. 2014. p. 153–167.

14

June 11, 2019

Advanced Robotics

adr2018a[70] Al-Shanoon AAS, Ahmad SA, b Hassan MK. Slip detection with accelerometer and tactile sensors ina robotic hand model. IOP Conference Series: Materials Science and Engineering. 2015;99(1):012001.
[71] Fernandez R, Payo I, Vazquez AS, Becedas J. Slip detection in a novel tactile force sensor. Cham:
Springer International Publishing. 2016. p. 237–252.
[72] Melchiorri C. Slip detection and control using tactile and force sensors. IEEE/ASME Transactions
on Mechatronics. 2000;5(3):235–243.
[73] Romano JM, Hsiao K, Niemeyer G, Chitta S, Kuchenbecker KJ. Human-inspired robotic grasp
control with tactile sensing. IEEE Transactions on Robotics. 2011;27(6):1067–1079.
[74] Holweg EGM, Hoeve H, Jongkind W, Marconi L, Melchiorri C, Bonivento C. Slip detection by tactile
sensors: algorithms and experimental results. In: Proceedings of IEEE international conference on
robotics and automation. Vol. 4. 1996. p. 3234–3239.
[75] Tsujiuchi N, Koizumi T, Ito A, Oshima H, Nojiri Y, Tsuchiya Y, Kurogi S. Slip detection with
distributed-type tactile sensor. In: 2004 IEEE/RSJ international conference on intelligent robots
and systems (iros). Vol. 1. 2004. p. 331–336.
[76] Gunji D, Mizoguchi Y, Teshigawara S, Ming A, Namiki A, Ishikawa M, , Shimojo M. Grasping
force control of multi-fingered robot hand based on slip detection using tactile sensor. In: 2008
IEEE international conference on robotics and automation. 2008. p. 2605–2610.
[77] Ho VA, Nagatani T, Noda A, Hirai S. What can be inferred from a tactile arrayed sensor inautonomous in-hand manipulation? In: 2012 IEEE international conference on automation scienceand engineering. 2012. p. 461–468.
[78] Hasegawa T, Honda K. Detection and measurement of fingertip slip in multi-fingered precision
manipulation with rolling contact. In: International conference on multisensor fusion and integration
for intelligent systems (mfi 2001). 2001. p. 43–48.
[79] Jamali N, Sammut C. Slip prediction using hidden markov models: Multidimensional sensor data
to symbolic temporal pattern learning. In: 2012 ieee international conference on robotics and automation. 2012.
[80] Reinecke J, Dietrich A, Schmidt F, Chalon M. Experimental comparison of slip detection strategies
by tactile sensing with the BioTac on the DLR hand arm system. In: 2014 IEEE international
conference on robotics and automation (icra). 2014. p. 2742–2748.
[81] Su Z, Hausman K, Chebotar Y, Molchanov A, Loeb GE, Sukhatme GS, Schaal S. Force estimationand slip detection/classification for grip control using a biomimetic tactile sensor. In: 2015 IEEERAS 15th international conference on humanoid robots (humanoids). 2015. p. 297–303.
[82] Veiga F, van Hoof H, Peters J, Hermans T. Stabilizing novel objects by learning to predict tactile
slip. In: 2015 IEEE/RSJ international conference on intelligent robots and systems. 2015. p. 5065–
5072.
[83] Veiga F, Peters J, Hermans T. Grip stabilization of novel objects using slip prediction. IEEE
Transactions on Haptics. 2018;.
[84] Tada M, Imai M, Ogasawara T. Development of simultaneous measurement system of incipient
slip and grip/load force. In: Proceedings 9th IEEE international workshop on robot and humaninteractive communication. IEEE RO-MAN 2000. 2000. p. 57–62.
[85] Ikeda A, Kurita Y, Ueda J, Matsumoto Y, Ogasawara T. Grip force control for an elastic finger
using vision-based incipient slip feedback. In: 2004 IEEE/RSJ international conference on intelligent
robots and systems (iros). Vol. 1. 2004. p. 810–815.
[86] Obinata G, Dutta A, Watanabe N, Moriyama N. Vision based tactile sensor using transparent elastic
fingertip for dexterous handling. In: Kolski S, editor. Mobile robots. chap. 7. Rijeka: IntechOpen.
2007.
[87] James JW, Pestell N, Lepora NF. Slip detection with a biomimetic tactile sensor. IEEE Robotics
and Automation Letters. 2018;3(4):3340–3346.
[88] Allen PK, Miller AT, Oh PY, Leibowitz BS. Integration of vision, force and tactile sensing for
grasping. Int J Intelligent Machines. 1999;4:129–149.
[89] Hsiao K, Chitta S, Ciocarlie M, Jones EG. Contact-reactive grasping of objects with partial shapeinformation. In: 2010 IEEE/RSJ international conference on intelligent robots and systems. 2010.
p. 1228–1235.
[90] Torres-Jara E, Natale L. Sensitive manipulation: Manipulation through tactile feedback. International Journal of Humanoid Robotics. 2018;15(01):1850012.
[91] Chuang C, Wang M, Yu Y, Mu C, Lu K, Lin C. Flexible tactile sensor for the grasping control ofrobot fingers. In: 2013 international conference on advanced robotics and intelligent systems. 2013.

15

June 11, 2019

Advanced Robotics

adr2018ap. 141–146.
[92] Venter J, Mazid AM. Tactile sensor based intelligent grasping system. In: 2017 IEEE international
conference on mechatronics (icm). 2017. p. 303–308.
[93] Yamaguchi A, Atkeson CG. Grasp adaptation control with finger vision: Verification with deformable and fragile objects. In: the 35th annual conference of the robotics society of japan (rsj2017).
2017. p. 1L3–01.
[94] Maria GD, Natale C, Pirozzi S. Slipping control through tactile sensing feedback. In: 2013 IEEE
international conference on robotics and automation. 2013. p. 3523–3528.
[95] Cirillo A, Cirillo P, Maria GD, Natale C, Pirozzi S. Modeling and calibration of a tactile sensor forrobust grasping. IFAC-PapersOnLine. 2017;50(1):6843 – 6850.
[96] Dang H, Allen PK. Stable grasping under pose uncertainty using tactile feedback. Autonomous
Robots. 2014;36(4):309–330.
[97] Li M, Bekiroglu Y, Kragic D, Billard A. Learning of grasp adaptation through experience andtactile sensing. In: 2014 IEEE/RSJ international conference on intelligent robots and systems.
2014. p. 3339–3346.
[98] Hyttinen E, Kragic D, Detry R. Estimating tactile data for adaptive grasping of novel objects. In:2017 IEEE-RAS 17th international conference on humanoid robotics (humanoids). 2017. p. 643–648.
[99] Calandra R, Owens A, Upadhyaya M, Yuan W, Lin J, Adelson EH, Levine S. The feeling of success:
Does touch sensing help predict grasp outcomes? PMLR. 2017;78:314–323.
[100] Calandra R, Owens A, Jayaraman D, Lin J, Yuan W, Malik J, Adelson EH, Levine S. More thana feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation
Letters. 2018;3(4):3300–3307.
[101] Hogan F, Bauza Villalonga M, Canal Anton O, Donlon E, Rodriguez A. Tactile regrasp: Grasp
adjustments via simulated tactile transformations. In: IEEE/RSJ international conference on intelligent robots and systems (iros). 2018.
[102] Guo D, Sun F, Fang B, Yang C, Xi N. Robotic grasping using visual and tactile sensing. Information
Sciences. 2017;417:274 – 286.
[103] Cramphorn L, Ward-Cherrier B, Lepora NF. Tactile manipulation with biomimetic active touch.
In: 2016 IEEE international conference on robotics and automation (icra). 2016. p. 123–129.
[104] Kaboli M, Yao K, Feng D, Cheng G. Tactile-based active object discrimination and target object
search in an unknown workspace. Autonomous Robots. 2018;.
[105] Murali A, Li Y, Gandhi D, Gupta A. Learning to grasp without seeing. ArXiv e-prints. 2018;(arXiv:1805.04201). 1805.04201.
[106] Ding Y, Bonse J, Andre R, Thomas U. In-hand grasping pose estimation using particle filtersin combination with haptic rendering models. International Journal of Humanoid Robotics. 2018;15(01):1850002.
[107] van Hoof H, Chen N, Karl M, van der Smagt P, Peters J. Stable reinforcement learning with
autoencoders for tactile and visual data. In: 2016 ieee/rsj international conference on intelligent
robots and systems (iros). 2016.
[108] Madry M, Bo L, Kragic D, Fox D. St-hmp: Unsupervised spatio-temporal feature learning for tactile
data. In: 2014 ieee international conference on robotics and automation (icra). 2014. p. 2262–2269.
[109] Sung J, Salisbury JK, Saxena A. Learning to represent haptic feedback for partially-observable
tasks. In: 2017 ieee international conference on robotics and automation (icra). 2017. p. 2802–2809.
[110] Sutanto G, Ratliff N, Sundaralingam B, Chebotar Y, Su Z, Handa A, Fox D. Learning latent space
dynamics for tactile servoing. In: Proceedings of the ieee international conference on robotics andautomation (icra) 2019. IEEE. 2019 May.
[111] Luo S, Bimbo J, Dahiya R, Liu H. Robotic tactile perception of object properties: A review.
Mechatronics. 2017;48:54–67.
[112] Liu H, Wu Y, Sun F, Guo D. Recent progress on tactile object recognition. International Journalof Advanced Robotic Systems. 2017;14(4).
[113] Hosoda K, Tada Y, Asada M. Anthropomorphic robotic soft fingertip with randomly distributed
receptors. Robotics and Autonomous Systems. 2006;54(2):104–109.
[114] Abdullah SC, Wada J, Ohka M, Yussof H. Object exploration using a three-axis tactile sensing
information. Journal of Computer Science. 2011;7(4):499–504.
[115] Fishel J, Loeb G. Bayesian exploration for intelligent identification of textures. Frontiers in Neurorobotics. 2012;6:4.
[116] Erickson Z, Chernova S, Kemp CC. Semi-supervised haptic material recognition for robots using

16June 11, 2019

Advanced Robotics

adr2018a

generative adversarial networks. In: Proceedings of the 1st annual conference on robot learning.
Vol. 78. 2017. p. 157–166.
[117] Funabashi S, Morikuni S, Geier A, Schmitz A, Ogasa S, Torno TP, Somlor S, Sugano S. Object
recognition through active sensing using a multi-fingered robot hand with 3d tactile sensors. In:
2018 ieee/rsj international conference on intelligent robots and systems (iros). 2018.
[118] Tomo TP, Schmitz A, Wong WK, Kristanto H, Somlor S, Hwang J, Jamone L, Sugano S. Coveringa robot fingertip with uskin: A soft electronic skin with distributed 3-axis force sensitive elements
for robot hands. IEEE Robotics and Automation Letters. 2018;3(1):124–131.
[119] Schneider A, Sturm J, Stachniss C, Reisert M, Burkhardt H, Burgard W. Object identification withtactile sensors using bag-of-features. In: 2009 ieee/rsj international conference on intelligent robots
and systems. 2009. p. 243–248.
[120] Yuan W, Mo Y, Wang S, Adelson EH. Active clothing material perception using tactile sensing
and deep learning. In: 2018 ieee international conference on robotics and automation (icra). 2018.
p. 1–8.
[121] Wang S, Wu J, Sun X, Yuan W, Freeman WT, Tenenbaum JB, Adelson EH. 3d shape perception from monocular vision, touch, and shape priors. In: 2018 ieee/rsj international conference on
intelligent robots and systems (iros). 2018.
[122] Rosales C, Spinelli F, Gabiccini M, Zito C, Wyatt JL. Gpatlasrrt: A local tactile exploration planner for recovering the shape of novel objects. International Journal of Humanoid Robotics. 2018;15(01):1850014.
[123] Ottenhaus S, Kaul L, Vahrenkamp N, Asfour T. Active tactile exploration based on cost-aware
information gain maximization. International Journal of Humanoid Robotics. 2018;15(01):1850015.
[124] Soft Robotics Toolkit. https://softroboticstoolkit.com/. 2018. [Online; accessed Oct-26-2018].
[125] Zhang Y, Kan Z, Tse YA, Yang Y, Wang MY. Fingervision tactile sensor design and slip detection
using convolutional lstm network. ArXiv e-prints. 2018;(arXiv:1810.02653). 1810.02653.

17