Evaluation of a Vision-based Tactile Sensor
Kazuto Kamiyama, Hiroyuki Kajimoto, Naoki Kawakami and Susumu Tachi
Graduate School of Information Science and Technology
The University of Tokyo, Tokyo, Japan
Email: {kazuto, kaji, kawakami, tachi}@star.t.u-tokyo.ac.jp

Abstract— Humans can perceive not only the magnitude but
also the direction of force applied on the fingertip. When we
grasp an object, the weight of it is felt through force that isparallel to the skin of the fingertip, which is how the object
can be grasped without slipping. Focusing on this point, we havedeveloped a tactile sensor that can measure a distribution of force
vectors. The measurement method is as follows. The tactile sensor
consists of a transparent elastic body, blue and red markers
inside the elastic body, and a color CCD camera. An applied
force on the elastic body results in movements of the markers,
which are acquired by the CCD camera. The distribution offorce vectors is calculated using this information. This paper
reports experimental evaluation results concerning accuracy ofdetermining position of markers, determination of magnitude anddirection of force, spatial resolution, and calculation timing.

I. I NTRODUCTION
We aquire environmental information through optical,
acoustic, tactile sensation and so on. Though it’s a well known
fact, only tactile sensation has different aspect whereas opticaland acoustic sensation aquire the external information. The
aspect is to generate consciousness that regions of body suchas arm and leg are part of ourselves. Usually, we are notconcious of it due to nothing special. However, becoming
paralysed on the arm by loss of blood flow, feeling as if thearm were not part of ourselves is arisen.
Just as tactile sensation is important for humans, it is quite
useful for robots as well. Robots equipped with a tactile sensor
are more capable of fine manipulation and avoiding criticalbehaviors. Providing tactile sensation to a robotic hand andfingertip is particularly necessary for delicate operation. As
an example, suppose we have the simple task of picking up
a fragile object on a table. When we execute this operation,
the weight of the object is felt not only by contraction of
the muscle but also force parallel to the skin of the fingertip.
The object is safely picked up by adjusting the proportionof perpendicular and parallel force on the skin. In a similarmanner, robots require sensing the force on the fingertip to
perform the operation dexterously.
To detect the weight of the object while grasping it, itis necessary to sense the force parallel to the surface of atactile sensor. This means that the tactile sensor needs tomeasure distribution of force vectors, but there are few sensors
that are able to measure such a thing[1]-[4]. Most of sensors
use a sensing unit in which electric resistance or capacitancedepends on force. To use such sensors with the intent tomeasure the distribution of force vectors, the wiring becomesquite complex and the size of the unit increases with theamount of necessary measured information. When the sensors
are built into a robot hand, this becomes a severe problem interms of miniaturizing and high-density sensing
An alternative tactile sensing method makes use of opticalinformation. In particular, sensors using visual images havegreat potential for decreasing the amount of wiring. There are
only a few known sensors that can measure distribution offorce vectors[5]. Ohoka et. al developed a method using anelastic sheet and a transparent board parallel to the sheet inwhich pyramidal protrusions are arranged. An applied force
makes the protrusions contact the sheet, and the force isobtained by measuring contact area[6]. Ohoka’s method uses
four protrusions to acquire a force vector, and is therefore
inadequate for high-density sensing. Ferrier et. al proposed atactile sensor that used markers inside an elastic body and afiber scope[7]. This sensor is formed into a fingertip shape andis successfully miniaturized. However, only a relatively small
amount of information is actually measured, and a complex
processing method is used to acquire direction of force, which
could potentially lead to inaccuracy and instability.
We focus on sensing methods that use visual images andhave proposed a vision-based tactile sensor[8]. This sensor canmeasure distribution of force vectors, is easily miniaturized,
and measures with high density.
II. T HEORY OF MEASURING DISTRIBUTION OF FORCEVECTORSThe tactile sensor which we have proposed uses a transparent elastic body and a color CCD camera. By photographinga certain marker on the interior of an elastic body by CCD,
when force is applied to the surface, the variation informationof the interior is measured, and is used to reconstruct a force
vector distribution.
Various methods can be considered to acquire variation
information. Our current approach is to measure the horizontal
movement of spherical markers in the elastic body, which
are located in an N × N array at a specified depth. To
gather sufficient information for the reconstruction of the force
vectors, we used two layers of markers that are located at
different depths (Fig.1). These layers could be distinguishedby the colors of the markers (red and blue). We set the x-yplane parallel to the sensor surface and the z axis extending
vertically on the interior. By measuring these markers from a
positive z direction with a CCD camera, we could obtain two
sets of two-dimensional movement vectors at different depths,

so the amount of information is increased and the distributionof force vectors can be readily obtained.

z

blue marker
red marker

z

Force

ux

=

uy

=

deformedby force

side view
yx

yx
movementof markers

top viewyFig. 1.

multiplying the whole equation by a constant, so it is set to 1
in the sequel.

yOverview of Image-based Sensing

A. Measuring the displacements of markers
To measure the movements of the markers, as preprocessing,
the position of markers is determined from the CCD image
when the elastic body doesn’t vary. This preprocessing allows
the markers to be located at arbitrary positions and eliminates
the need for camera positioning. To pinpoint the marker center
position, a center of mass measurement method is applied.
Herewith, the center position, which was originally an integer
value based on a pixel unit, becomes a real number with subpixel accuracy.
When force is applied to the elastic body surface, themarkers will move, and this movement is observed by the
CCD. The picture is subdivided into smaller squares, eachof which is centered on the marker position measured in theinitial state that force is not applied. By taking the differenceof the position of the center in each square of the picture
before and after the movement, the markers’ translation in the
xy-direction can be calculated. If the movement falls outsideof the divided square, a so-called ’tracking’ method can beapplied, which only uses two sequential images.
B. Obtaining distribution of force vectors from displacement
vectors
In order to obtain force vectors from movement vectors, we
use the theory of elasticity [9] by assuming that the elastic
body is half-spaced, uniform and has linearity. We set the zaxis perpendicular to the elastic body’s interior surface, and thexy-plane parallel to the surface of the elastic body. Following
Eq. 2 and Eq. 1, we express the movement vector u = (ux , uy )of the interior point r = (x, y, z) within a plane parallel to thexy-plane when a force vector f = (fx , fy , fz ) is applied tothe surface of the elastic body. Here, σ is a Poisson ratio,
which is set to 0.5, by assuming that the elastic body isideally incompressible. E is Young’s modulus and must be
appropriately defined according to the actual elastic body used,
but from this equation it is apparent that E is in effect only



xz
1+σ
(1 − 2σ)x2(1 − σ)r + z
fx
−
fz +
2πE
r3
r(r + z)r(r + z)[2r(σr + z) + z 2 ]x
+
(xf
+
yf
)
,
(1)
x
y
r3 (r + z)2


yz
1+σ
(1 − 2σ)y2(1 − σ)r + z
fy
−
fz +
3
2πE
r
r(r + z)r(r + z)[2r(σr + z) + z 2 ]y+(xfx + yfy ) .
(2)
r3 (r + z)2

From these equations, when an unit force f = (fx , fy , fz )is applied in each direction x,y,z, i.e. (1, 0, 0), (0, 1, 0),(0, 0, 1), the displacement vector of the point in the planeat certain depth z = z1 is calculated. We represent thisvector by uf x = (hxx1 , hyx1 ), uf y = (hxy1 , hyy1 ), uf z =(hxz1 , hyz1 ). These displacement h can be considered as
an impulsive response to unit force of each direction fromthe origin. When the force applied to the surface of theelastic body is reconsidered as a vector distribution expressedas f(x, y) = (fx (x, y), fy (x, y), fz (x, y)), the displacement
vector m
 1 (x, y) = (mx1 (x, y), my1 (x, y), mz1 (x, y)) for eachpoint (x, y) in the plane at a depth z1 and is calculated in theform of a convolution, denoted by an asterisk (Eq.3). Notethat this calculation utilizes the assumption of linearity.
mx1 (x, y) =my1 (x, y) =hxx1 ∗ fx + hxy1 ∗ fy + hxz1 ∗ fz ,hyx1 ∗ fx + hyy1 ∗ fy + hyz1 ∗ fz (3)A discrete form of this calculation can be expressed asa matrix representation. The displacement vectors and force
vectors are sampled at M × N points, and the x, y, z
components of the displacement vector are expressed as Mx1 ,My1 , Mz1 in the matrix. Elements from (1, 1) to (M, N ) arerenumbered using one suffix and are reinserted as a vector.
Stress vectors are inserted in the same fashion. By doing so,
Eq.3 rewritten in matrix form (Eq.4).


Mx1
My1




=

Hxx1
Hyx1

Hxy1
Hyy1

Hxz1
Hyz1






Fx
 Fy 
Fz(4)

Since this equation calculates displacement of an interior
point when a force vector distribution is applied to the surfaceof the elastic body, its inverse becomes a formula whichcalculates the force vector distribution by given the measured
value M . This is the measurement principle of this research.
At this point, consider the number of M vector elements on
the left side of the equation and the F vector elements on theright side. Since the number of sampling points is M × N , thenumber of elements of a displacement vector is M ×N ×2 andthe number of elements of a force vector is M ×N ×3, there are
more unknowns than equations. Thus, the measuring technique
is applied to another height by using color information. Asthe impulse response varies according to depth, we obtain thefollowing equation (Eq.5).
 
Hxx1
Mx1
 My1   Hyx1
  Mx2  =  Hxx2
My2
Hyx2


Hxy1
Hyy1
Hxy2
Hyy2



Hxz1 FxHyz1  Fy 
Hxz2 
Fz
Hyz2

(5)

Eq. 5 is simply described as M = HF . This becomes asituation with more equations than unknowns, so it is easy to
calculate a force vector distribution F . Since the matrix H is
not a square matrix, it does not have an inverse matrix, so
force will be calculated using a pseudo-inverse matrix[10].
III. D EVICE OVERVIEW
A system diagram of the sensor used for evaluation is shownin Fig.2.
spherical markers,
1.5mm interval

Fig. 3.

Captured image by CCD camera

IV. E VALUATION E XPERIMENT
A. Measuring accuracy of movement vectors

force

transparent
elastic bodyacrylic
board

When a force vartical to the sensor surface is applied witha cylinder (5mm diameter), the observed displacement vectorsof red markers are as shown in Fig.4. The magnitude of
the force is 910gf. Radial variation centered around force
application point is generated, and observed displacement
vectors is obtained by projection to x−y plane. The lengths ofthe vectors in Fig.4 are five times as long as real displacement.
y(pixel)CCD camera

Fig. 2.

Image of sensor system

The transparent elastic body has a thickness of 40mm, a
length of 90mm, a width of 100mm, and has an opaque
black shading layer prepared on its surface. Blue markers
are placed at a depth of 3mm, and red markers at a depth
of 6mm. We use plastic balls (DAICEL FINECHEM, LTD.
FREEPLASTIC) with a diameter of 0.6mm as color markers,
arranged in approximately 1.5mm intervals, which is nearly thehuman’s limit of spatial resolution for discriminating between
two separated points.
The markers inside the elastic body are photographedthrough the transparent acrylic board that fixates the elastic
body. One pixel corresponds to about a 0.05mm square andthe size of the photographed area is about 18mm×14mm. The
captured image is output in NTSC format and sent to a PC
through a capture unit with a USB connection.
Fig. 3 is photographed image by the CCD camera. Blue andRed markers can be observed at a regular interval. Using thisimage, The force vectors is calculated.

250

200

150

100

50

0
0

50

100

150

200

250

300

x(pixel)Fig. 4. Calculated displacement vectors of red markers when applied a zdirectional force

As a first evaluation experiment, we investigated the accuracy of the above-mentioned method for displacement vector
measurement. Two images of the markers are captured, thefirst when no force was applied, and the second which wasdisplaced 3 pixels in the x-direction. Displacement vectors ofthe markers were calculated using these two images. The meanx and y displacements of red and blue markers are presented
in Table 1. It is apparent that the displacements are properlymeasured, and with a standard deviation of less than 0.1 pixels,
which corresponds to 50µm, indicating considerable accuracy.
TABLE IT EST OF M EASURING MOVEMEMT
ComponentBlueRed

direction
+x
+y
+x
+y

Average(pixel)
2.98
0.00
3.00
0.00

SD
0.06
0.06
0.09
0.10

B. Evaluation of calculated force vectors
Fig.5(a) shows the distribution of three dimensional force
vectors when a force vertical to the surface was applied using acylinder of 5mm diameter, and Fig.5(b) shows the distributionof force parallel to the surface using a cylinder of 12mm. In
this section, these measured force vectors are evaluated.

z

18.7
y[mm]

9.4

11.7

23.4
x[mm]

0
(a)Three dimensional force vectors
when force is applied z-direction

1) Effectiveness of two layers of markers: We simulate theeffectiveness of the two layers of markers and the influenceof distance between the blue and red layer.
Given a force distribution, displacement vectors are calculated using Eq.5. Marker positions of the simulation were
derived from an actual CCD image. An error was added tothese movement vectors with a mean and standard deviation of
0pixel and 0.1pixel, respectively. Using the error added to thedisplacement vectors, we then calculate distribution of force.
The effect of two layers is investigated by comparing between
calculated and actual force vectors. This simulation is iterated
as the given distance between the two layers is changed. The
numerical value used in this evaluation is the mean difference
between calculated and actual force obtained in each direction
x, y, and z.
Number of sampling points is 10 × 8 points. A certain
sampling point located at xf in the x-axis position and yf
in the y-axis position is represented as (xf , yf ). The given
force is equally applied on an area surrounded by the points(4, 3), (7, 3), (4, 6), (7, 6), with a magnitude of 100gf in thez-direction and 50gf in the y-direction.
Fig.6 shows the difference between the actual force and thecalculated force obtained from the simulation. The value of thehorizontal axis means, starting from the left, the case of only
the blue layer, the case of blue and red marker layer at the same
depth, each case with a distance between the two layers from
1mm to 6mm, then 8mm and 13mm. The case that the blue andred layers are located at same depth is equivalent to the casethat the density of sampling points at only one layer is doubled.
Though the average of the difference is almost the same in thefirst two cases, the more the layers are separated, the less themean of the difference becomes. However, this trend bottoms
out, and with much more separation, the mean difference
increases. This conclusion suggests that the difference between
the two layers is preferably about 2mm or 3mm, which was
the distance adopted in our tactile sensor.
1.1

difference
Fz
Fx
Fy

1.0
normalized difference

position y[mm]

18.7

9.4

0

11.7
position x[mm]23.4(b)force vectors parallel to the surface
when torqued force is applied
Fig. 5.

0.9
0.8
0.7
0.6
0.5
0.4
0.3
one
layer

Measured distribution of force vectors
Fig. 6.

0(doubledensity)

1

2

3

4

5

6

8

13

distance between blue and red layer [mm]Dependence of the error on the distance between two layers

Measured force [gf]

800

600

400

200
Measured value
Line the slope of which is 1
0

Fig. 7.

0

200

400
600
Applied force [gf](measured by 6-axis force sensor)

800

Relation between applied and measured force of z-direction

Next, we estimated the resolution of angle of the force. The
magnitude of the applied force is fixed at 400gf, and the force
is applied at an angle respective to the normal to the surface.
This angle changes from 0deg to 45deg by 5deg increments.
A cylinder with a diameter of 5mm is used as a probe to
apply force. The base plane of the probe is kept parallel tothe xy-plane.
The experimental result is shown in Fig.8. It is apparentthat
the proposed tactile sensor measures the angle of force precisely and linearly. The data spread of this graph shows thatthe resolution of the angle of measured force is about 5deg.

Angle of measured force [deg]2) Resolution of magnitude and direction: In this section,
we evaluate at first the resolution of the force magnitude, andthen the resolution of direction. Each directional force Fx , Fy ,Fz is collected in 1-dimensional vector, and the summationnmsum=
of force is calculated as Fx sum =k=1 Fx (k), Fy
nm
nm
sum
F
(k),
F
=
F
(k).
By
comparing
between
y
z
z
k=1
k=1
this summation F sum and the actual applied force, we examine
the resolution and linearity of measured force vectors. The
actually applied force is measured by a 6-axis force sensor
and controled by a xyz-stage.
First, we investigated magnitude of the measured force.
Increasing the applied z-directional force from 0gf to 750gf
by 50gf increments, the value acquired from the sensor isrecorded, and this procedure is repeated 10 trials. The force
is applied using a probe, the contact site of which formed
5mm diameter hemisphere. Fig.7 shows the result of this experiment. This graph indicates that the sensor can sufficiently
measure the force linearly and a magnitude resolution of about
30gf. The elasticity of the sensor used in this experiment is 10
times harder than that of the human fingertip. By making theelasticity of the sensor as soft as the fingertip, the resolution
improves to about 3gf.

40
Magnitude of Applied force
is fixed at 400gf
30

20

10
Measured value
Line the slope of which is 1

0
0

10
20
30
40
Angle of applied force [deg](measured by 6-axis force sensor)

Fig. 8. Relation between applied and measured force with an angle fromthe surface3) Spatical resolution of the distribution of forces: The
proposed tactile sensor calculates force by using a pseudoinverse matrix, so the calculated result for each point includes
an error and is distributed around its neighbors. Thus, in thissection we investigate the effects of this error to the spatial
resolution of the forces.
The evaluation uses a force distribution acquired by applying force at a certain point on the surface of the elastic body.
However, the sampling interval is not fine enough to estimate
the distributed error. Therefore we take a sequence explained
as follows.
For simplicity, the force vector is assumed to be twodimensional. Two adjacent sampling points from the distribution of forces were selected. The distance between these twopoints was subdivided into n points, and a given z-directional
force was applied at each point. Then, the calculation of force
vectors at each point was executed, with each point regarded
as the origin. A graph is constructed with the horizontal axis
the distance from each origin, and the vertical axis as the zcomponent of the calculated force vector.
By using the method explained above, the density of force
sampling points is increased. Then, measuring the slope of theobtained force distribution led to the spatial resolution of force
distribution.
Fig.9 shows the result of the experiment. The value of thevartical axis is normalized using the maximum value among
the acquired data. The magnitude of applied force directed in
the z-direction is fixed at 200gf and the adjacent sampling
points are located at the center of the area in which force
distribution is acquired. The actual sampling interval is 2.2mm
equal to that Fig.5 and theadjacent points are subdevided into
0.37mm intervals.

From the graph in Fig.9, it is apparent that the half value ofwidth is about 4mm. That is to say, the spatial resolution of
the forces is estimated at about 4mm. The obtained resolution
is equal to one lead from the sampling interval, therefore theproposed sensor is proven to have sufficient spatial resolution.
force applied point(x[mm],y[mm])Normalized force Fz

1.0(11.1, 8.9)
(11.8, 8.9)
(12.2, 8.9)
(12.6, 8.9)
(13.0, 8.9)
(13.3, 8.9)

0.8
0.6
0.4
0.2
0.0

-0.2
-10 -8-6 -4 -2 0 2 4 6 8 10 12Distance from force applied point x[mm]Fig. 9. z-component of measured force at high density sampling when point
force is applied

R EFERENCES

C. Estimation of Computational time
In this section, we estimate computation time from theinstant image data is loaded into main memory of the PC until
the force vector data is obtained. For reference, the time fromobtaining data until displaying this data like Fig.5 is also indicated. The computational time is acquired using timer based
CPU frequency. The speed of the processor used in the PC for
this benchmark, is 1.8GHz (Intel(R), Pentium4(R) processor)
and the program runs under Windows 2000. Even though
the fluctuation of the processor frequency was 100MHz, thismethod had an accuracy of 10µsec. Furthermore, by acquiring
computation time more than once, the effects of other programinterruptions can be decreased and the estimation is made more
precise.
The computation time is measured 300 times and theaverage is shown in Table tab:time. It is apparent from thistable that the computation time to obtain the force vectors ison the order of milliseconds and therefore suitable for realtime applications.
TABLE IICOMPUTATION TIME UNTIL THE DATA OF FORCE VECTORS IS OBTAINED

time[msec]

time until obtain the forces
2.9

vectors. The outcome of the experiments shows a 50µm accuracy of determining displacement vectors, a 30gf magnituderesolution, and a 5deg angle resolution of calculated force. The
particular merit of this measurement principle of the sensor isthat it is simple to photograph the markers in the elastic body;therefore it is easy to improve the sensitivity of the sensor byincreasing the density of markers and changing the resolutionof the color CCD camera.
Because of frequent contact to an object in practice, a tactile
sensor must be durable. The proposed sensor is structured
so that the CCD camera as an actual measurement device isseparated from the contact point, in contrast to from a sensor
in which sensing units are implanted. Hence, if the elastic
portion of the sensor breaks, it is relatively easy to replace.
Additionally, the tactile sensor we have developed can measuredisplacement vectors for an arbitrary shape, it can easily be
mounted on a robot hand in the shape of a fingertip.
To append the human-like sense of touch to robots, atactile sensor needs to measure distribution of force vectors,
must be readily miniaturized, and capable of high-density
measurement. Through our evaluation experiments, we haveverified that the tactile sensor we have proposed fulfills thesedemands with high performance. At present, we are developing
a finger shaped tactile sensor with the aim to mount it on a
robot hand.

displaying time
1.9

V. C ONCLUSION
In this paper, we reported the evaluation of the proposed
tactile sensor, which can measure the distribution of force[1] R.S.Fearing, ”Using a Cylindrical Tactile Seosor for Determining Curvature”, IEEE Trans. on Robotics and Automation, vol.7, No.6, pp.806817, 1991.
[2] J. G. Silva, A. A. Carvalho, D. D. Silva, ”A Strain Gage Tactile
Sensor for Finger-Mounted Applications”, IEEE Trans. on Instruction
and Measurement, vol.51, No.1, pp.18-22, 1991.
[3] H. Shinoda, K. Matsumoto, S. Ando, ”Acoustic Resonant Tensor Cellfor Tactile Sensing”, Proc. IEEE International Conference on Robotics
and Automation, pp.3087-3092, 1997.
[4] D. T. V. Pawluk, J. S. Son, P. S. Wellman, W. J. Peine, R. D. Howe,”Distributed Pressure Sensor for Biomechanical Measurements”, Journalof Biomechanical Engineering, Transactions of the ASME, Vol.120,
No.2, pp. 302-305, 1998.
[5] M.Ohoka, Y.Mituya, K.Hattori, I.Higashioka, ”Data conversion capability of optical tactile sensor featuring an array of pyramidal projections”,
IEEE International Conference on Multisensor Fusion and Integration
for Intelligent Systems, pp.573-580, 1996.
[6] K.Tanie, K.Komoriya, M. Kaneko, S.Tachi, ”A HIGH RESOLUTIONTACTILE SENSOR”, Proceedings of the 4th International Conferenceon Robot Vision and Sensory Controls, pp.251-261, 1987.
[7] N. J. Ferrier, R. W. Brockett, ”Reconstructing the Shape of a Deformable
Membrane from Image Data”, The International Journal of Robotics
Research, Vol.19, No.9, pp.795-816, 2000.
[8] K. Kamiyama, H. Kajimoto, M. Inami, N. Kawakami, S.Tachi, ”A
Vision-based Tactile Sensor”, The 11th International Conference on
Artificial reality and Telexistence, 2001.
[9] L. D. Landau, E. M. Lifshitz, ”Theory of Elasticity”, BUTTERWORTHHEINEMANN, 1985.
[10] W. Menke, ”Geophysical Data Analysis: Discrete Inverse Theory”,
Academic Press, Inc., 1989.
[11] R. S. Fearing, J. M. Hollerbach, ”Basic Solid Mechanics for
Tactile Sensing”, The International Journal of Robotics Research,
Vol.4,No.3,pp.40-54, 1985.