2008 IEEE/RSJ International Conference on Intelligent Robots and Systems
Acropolis Convention Center
Nice, France, Sept, 22-26, 2008

Measurement of Force Vector Field of Robotic Finger using
Vision-based Haptic Sensor
Katsunari Sato, Kazuto Kamiyama, Hideaki Nii, Naoki Kawakami, and Susumu Tachi,
The University of Tokyo, Japan

Abstract—It is expected that the use of haptic sensors tomeasure the magnitude, direction, and distribution of a force will
enable a robotic hand to perform dexterous manipulations.
Therefore, we have developed a new type of finger-shaped haptic
sensor that can measure a three-dimensional force vector fieldover a contact surface. The sensor consists of a transparent
elastic body, two layers of internal blue and red markers, and a
CCD camera to capture the movements of the markers. Using
the elastic theory, we can calculate the force vector field from thecaptured movements of the markers. However, the elastic theory
cannot be applied to the finger-shaped sensor because of its
complicated shape. Therefore, we use actual measurements forthe calibration in order to develop a prototype of the sensor.
Then, we evaluate its basic performance. The result shows thatthe sensor performance can be improved further, and the sensor
can be successfully used in a robotic hand.

the magnitude, direction, and distribution of an applied force.
Conventional sensors mainly measure only one or two of theseparameters. However, all three parameters are essential forstable grasping and dexterous manipulation. Second, the sizeof the sensor should be small. In order to set many haptic
sensors in a humanoid robot hand, it is preferable to use
compact sensors. Finally, the sensor should be ahigh-performance device. In order to obtain accurate haptic
information, high magnitude, spatial, and temporal resolutions
are better.

I. INTRODUCTION
HAPTIC information acquired through the fingertips isimportant for human since it allows them to manipulate an
object dexterously. The object can be grasped in a stablemanner using haptic information such as the reflective force,
frictional force, and point of contact with the object. Robotic
systems use a similar methodology. In order to pick up, move,or lay down an object, a robotic hand requires haptic
information. Therefore, robotic hands require haptic sensors
to manipulate objects dexterously. Recently, two principal
types of haptic sensors have been invented for robotic hands[1] [2]: a force vector sensor that measures three- or
six-dimensional force at a point and a force field sensor thatmeasures the distribution of a one-dimensional force.
The objective of our research is to achieve dexterous
manipulation in various activities such as rolling clay between
the fingertips by using a robotic hand with a haptic sensor (Fig.
1). We have laid down three requirements for the haptic sensor
of the robotic hand. First, the sensor should be able to measure
Katsunari Sato, Hideaki Nii, Naoki Kawakami, and Susumu Tachi areaffiliated with the Graduate School of Information and Technology, The
University of Tokyo, Japan;;e-mail: (Katsunari_Sato,Hideaki_Nii)@ipc.i.utokyo.ac.jp,(kawakami, tachi)@star.t.u-tokyo.ac.jp.
Kazuto Kamiyama is affiliated with the Research Center for Advanced
Science and Technology, The University of Tokyo, Japan; e-mail:kazuto@hal. rcast.u-tokyo.ac.jp.
This research was financed by Nitta Corporation.

978-1-4244-2058-2/08/$25.00 ©2008 IEEE.

Fig. 1. Example of manipulation by robotic hand using our fingertip sensor.
The blue and green surfaces shown in the laptop display represent the
information measured by the sensors mounted on the index finger and thethumb of the robotic hand, respectively.

The first requirement can be satisfied by integrating many
force vector sensor elements [3] [4]. However, the use ofmany sensors can complicate the sensor structure and decrease
the sensor performance. Ferrier et. al. have proposed anotherapproach to satisfy the first requirement [5]. They havedeveloped a small tactile sensor to measure the force vector
field. However, this sensor requires a complex processing
method to acquire the direction of force, leading to inaccuracy
and instability.
In order to satisfy all the abovementioned requirements, wehave created a finger-shaped haptic sensor that can measure athree-dimensional force vector field applied on its surface.
The technology used by this sensor is the same as that used in
the vision-based haptic sensor developed by us [6]. This488

sensor has a simple structure; hence, it satisfies the second andthird requirements, i.e., having small size and high resolution.
In this paper, we present the measurement theory,
configuration, and calibration method of the finger-shaped
haptic sensor. In addition, we evaluate its accuracy when it isused as a force vector sensor and a force field sensor.

In this equation, u indicates the two-dimensional movement
vectors of the red and blue markers captured by the camera, f is
the three-dimensional force vector, and H is the conversion
matrix. The elements of H are calculated by using the elastic
theory. We obtain f from u by solving the inverse problem [8]of equation (1).

(f = HTH

II. FINGER-SHAPED HAPTIC SENSOR
A. Theory for measurement of force vector field
In order to enable the finger-shaped haptic sensor tomeasure the force vector field, we have applied the technology
used in the vision-based haptic sensor. This sensor consists ofa transparent elastic body with two layers of a marker matrix
and a color CCD camera. When a force is applied to thesurface, the markers move as shown in Fig. 2. The movementsof the markers are captured by the camera. We calculate thecenter of the markers before and after application of the force.
Therefore, we can record the movements of markers andcalculate the applied force using the elastic theory [7].

force

z

z

side
view

y defor-ymation x

x
top
view

y

y

Fig. 2. Movements of markers in elastic sensor body caused by force applied
to sensor surface [6].

We use two suppositions to calculate the force vector fieldfrom the movements of the markers. The first supposition isthat the sensor is a semi-infinite elastic body. The second
supposition is that the deformation of the elastic body is linear.
On the basis of these suppositions, the relationship between
the movement of marker and force is given by equation (1) as
follows:u = Hf .
⎡hRxx , hRyx , hRzx ⎤⎡u Rx ⎤⎢⎥
⎡ fx ⎤⎢u ⎥Ry ⎥⎢hRxy , hRyy , hRzy ⎥⎢,H = ⎢u=, f = ⎢⎢ f y ⎥⎥⎢u Bx ⎥
hBxx , hByx , hBzx ⎥⎢⎥⎢⎣ f z ⎥⎦⎢ ⎥⎢⎣hBxy , hByy , hBzy ⎥⎦⎢⎣u By ⎥⎦(1)) H u.
−1T(2)

It should be noted that the vision-based sensor uses a single
camera to capture the movements of the two layers of markers.
In order to calculate the three-dimensional force vector, it isnecessary to measure the movements of the markers in athree-dimensional manner. A stereo camera is suitable for
capturing the movements of the markers. However, the use ofa stereo camera increases the size of the sensor and the
difficulty in calibrating the camera. Hence, to overcome these
problems, we use two layers of markers and one camera.
This vision-based haptic sensor does not require any
electronic components such as resistors or capacitors. Hence,
it is easy to adapt the shape and size of the sensor to that of thefingertip. In addition, further developments in camera
technology will result in improved spatial and temporal
resolution of the sensor. The elastic body of the sensor
provides an added advantage. The elasticity enables the sensor
to appropriately measure the haptic information by
reproducing the physical interaction involved in the sense of
touch. Furthermore, the sensor is highly durable.
B. Design of sensor
We construct the fingertip sensor for the robotic hand by
applying vision-based technology.
The configuration of the sensor is shown in Fig. 3. The
elastic body and CCD camera are attached to the plastic baseof the robotic hand. The elastic body is made from transparent
silicon rubber. The shape of the elastic body is partly sphericaland partly cylindrical; hence, it resembles a human fingertip.
There are blue and red markers on the surface of the elastic
body. Because the sensor uses vision-based technology, lightfrom the environment produces noise. Therefore, the surfaceof the sensor is covered by a black sheet made from the siliconrubber. In addition, to capture the marker movements clearly,
they are illuminated by an LED mounted near the camera. We
use a camera with a wide-angle lens so that the camera cancapture all the markers.
C. Calibration method
In conventional vision-based haptic sensors, the applied
force is calculated from the movement of the markers using
equation (1). However, equation (1) has been derived on thebasis of two suppositions. These suppositions are notapplicable to the finger-shaped haptic sensor. Hence, weestimate the elements of the conversion matrix H from actual489

measurements.
Figure 4 shows the algorithm for the construction of H. First,
a force is applied on the surface of the sensor along three
directions; the components of the force along the three
directions are denoted by fx, fy, and fz. Then, we measure theforce elements caused by applying the force as follows:⎡ f xx , f xy , f xz ⎤⎥⎢f n = ⎢ f yx , f yy , f yz ⎥⎢f , f , f ⎥⎣ zx zy zz ⎦Figure 5 shows the apparatus used for the construction of
matrix H. We push the sensor surface by using a probe. The
probe edge is spherical and the diameter is 3.0 mm. The probe
is mounted on a six-axis force sensor to measure themagnitude of the applied force. Further, we use an xyz stage tocontrol the applied force. We first apply a force along thez-direction and then apply suitable forces along the x- andy-directions.

(3)
six-axis
force sensor
probe

where fmn indicates the m-directional element of force whenn-directional force is applied. Next, we capture themovements of the markers un. The x-directional marker
movements of a marker are given by equation (4).

[u n = u xx , u xy , u xz]

hn = u n f n(h = [h , h , h ])

−1

n

x

y

z

(5)

CCD camera
base

LED
red and blue
marker matrix

transparent elastic bodyFig. 3. Configuration diagram of finger-shaped vision-based haptic sensor.

f2

f3

1. Application of known
force fn to surface
fN

f1
elastic body
⎤
⎡
⎥
⎢
h
2
h
1
h
N
H=⎢
⎥
⎥⎦
⎢⎣
3. Constructionof matrix H

z(4)

where umn indicates the m-directional movement of marker
when n-direction force is applied. Using fn and un, we cancalculate the elements of matrix H as follows:

2. Measurement ofmarker movement un

Fig. 4. Algorithm for construction of conversion matrix H. The elements of H
are calculated from the measured force fn and the movements of the markers
denoted by un.

finger-shaped
haptic sensorx

jig forxyz-stageyFig. 5. Image of apparatus used for construction of conversion matrix H.

D. Construction of prototype of finger-shaped haptic
sensorWe have constructed a prototype of the finger-shaped
haptic sensor. The dimensions of the transparent elastic bodyof the prototype are 16 × 16 × 10 mm³. The elastic body ismade from silicon rubber (Shin-Etsu Chemical Co., KE-109).
The camera is a custom-made one comprising a video
compression unit and four image capturing units (ViewPLUS
Inc.). The resolution of each image is 640 × 480 pixels. The
centers of the markers are tracked with sub-pixel accuracy.
The frame rate of the camera is 60 fps. Figure 6 shows thecaptured images of the markers. To capture the markers
clearly, their diameters are set to 0.5 mm (approximately 15
pixels). Moreover, in order to prevent the overlapping of the
red and blue markers, they are separated by a distance of 2.5
mm (Fig. 6, left). The red and blue markers are placed at depthof 0.5 and 2.0 mm, respectively, from the surface of the sensor.
We print the markers using the silk-screen printmaking
technique. The markers are made from silicon rubber and acolorant.
The calibrated positions of the forces are shown in Fig. 6(right). In the prototype, the markers are separated by adistance of 2.5 mm. Therefore, we apply a force at 25 points
on the surface of the sensor. The points are also separated by adistance of 2.5 mm. The magnitude of the applied force isapproximately 100 gf.
Figure 7 shows the constructed prototype and the calculated
force vector field. When the sensor surface comes in contact
with an object, it measures the corresponding force vector
fields.

490

magnitude along the three directions: x, y, and z. Eachdirectional force, fx, fy, and fz, is represented by a
one-dimensional force vector and the summations of the force
denoted by Fx, Fy, and Fz, respectively, are calculated asshown in equation (6).

sampling point
2.5 mmx
2.5mm

2.5 mm
2.5mm

Fx = ∑ k =1 f x (k )
n

y

(6)

F y = ∑ k =1 f y (k )nFig. 6. Images of markers captured by camera (left) and points used forsampling force (right).

Fig. 7. Images of constructed sensor (left) and captured force vector field(right).

III. EVALUATION
The finger-shaped haptic sensor can be used both as a force
vector sensor and as a force field sensor. Therefore, we
evaluate the basic performance of the constructed prototype
using the following four criteria: the accuracy of measurement,
the resolution of the force magnitude when it is used as a force
vector sensor, the spatial resolution when it is used as a force
field sensor, and the time required for calculation. The
apparatus used in the evaluation of the environment is thesame as that used for calibration (Fig. 6).
A. Measurement accuracy
The force vector field is calculated from the movements ofthe centers of the markers captured by the camera. Therefore,
an error that occurs while capturing the movements of themarkers may produce noise in the measurement of the force.
In this section, we evaluate the accuracy of the measurementof the movements of the markers when no force is applied tothe surface of the sensor.
We capture two images of the markers when no force isapplied. Then, we displace the markers in one of the images by
5 pixels along the x-direction. The displacement vectors of themarkers are calculated using these two images. Table I shows
the average and the standard deviation of the x- andy-displacements of the red and blue markers. The result showsthat the movements of the markers are measured with subpixel
accuracy.

Fz = ∑ k =1 f z (k )n

We examine the resolution and linearity of the measured force
vectors by comparing the measured force F with the applied
force. The applied force is measured by using the six-axis
force sensor and the controlled xyz stage. In this evaluation,
we apply a force using a cylindrical probe with a diameter of
10 mm.
To investigate the resolution of magnitude in the x- andy-direction, we first applied a force of 200 gf along thez-direction. We then applied the force along the x- or
y-direction with magnitudes in the range of 20-200 gf byincreasing in steps of 20 gf. To investigate the resolution along
the z-direction, we applied a force along the z-direction withmagnitudes in the range of 20-200 gf by increasing in steps of
20 gf. The values obtained from the sensor are recorded. The
measurement procedure is repeated 5 times for each direction.
The graphs shown in Fig. 8 present the results of theevaluation. The horizontal and vertical axes represent theapplied force and the measured force, respectively. The dotsshow the results of each trial. From these results, it is observedthat the maximum resolution of magnitude is 30 gf in eachdirection. Thus, the linearity of the sensor is confirmed.
TABLE IMEASUREMENT OF MARKER MOVEMENT
MarkerRed
Blue

Direction
+x
+y
+x
+y

Average (pixel)
4.87
0.03
4.95
0.00

Std. Deviation (pixel)
0.14
0.14
0.13
0.10

C. Spatial resolution
The finger-shaped haptic sensor can be used to measure aforce field. In this section, we evaluate the spatial resolution of
the sensor. For the evaluation, we obtain the magnitude ofeach force vector when a force is applied at a certain point on
the surface of the elastic body. The magnitude of the force
vector f is calculated as shown in equation (7).

B. Resolution of magnitude
The finger-shaped haptic sensor can be used as a force
vector sensor. We evaluate the resolution of the force

491

f =

2

2

fx + f y + fz

2

(7)

is constructed by plotting the distance from each origin and thecalculated force magnitude on the horizontal and vertical axes,
respectively. By using the method explained above, thedensity of the points in the sampling region is increased. The
spatial resolution of the force distribution can be obtained
from the slope of the graph.
The magnitude of the force applied in the z-direction isfixed at 200 gf. The diameter of the probe used to apply theforce is 3.0 mm. The adjacent sampling points are located at
the center and the edge of the area in which the force field ismeasured. The actual sampling interval is 2.5 mm, and the
distance between adjacent points is subdivided into intervalsof 0.5 mm. We evaluate the spatial resolution at both thecenter and the edge of the sensor surface and in both the x- andy-directions.
Figure 9 shows the results of the experiment. The first graphshows the results obtained by applying a force at the center ofthe sensor. The second graph shows the results obtained byapplying a force at the edge of the sensor. The values plottedon the vertical axis are normalized using the maximum valueof the force magnitude. The dots in the graphs represent the
points at which the force is applied (x [mm], y [mm]).
From the graphs in Fig. 9, it can be observed that the full
width half maximum is approximately 5 mm. Therefore, thespatial resolution of the force is estimated to be approximately
5.0 mm. These results are identical to those obtained in theevaluations at other positions and directions. Therefore, thecomplicated shape of the sensor does not affect the spatial
resolution. The discontinuous result around the central
position is caused by an error in the calculation of thepseudo-inverse matrix in equation (2).

Measured force Fx [gf]

300
250
200
150
100
50
0
0

50

100

150

200

250

200

250

Applied force Fx [gf]Measured force Fy [gf]

300
250
200
150
100
50
0
0

50

100

150

Applied force Fy [gf]Measured force Fz [gf]

300
250
200
150
100
50
0
0

50

100

150

200

250

Applied force Fz [gf]Fig. 8. Relation between applied force and measured force in x-direction (top),
y-direction (middle), and z-direction (bottom).

It should be noted that the sampling interval is not
sufficiently small to evaluate the spatial resolution. Therefore,
we adopt the following procedure to evaluate the spatial
resolution. Two adjacent sampling points on the sensor
surface are selected. The distance between these two points is
subdivided into n points, and a known force is applied in thez-direction at each point. Then, the force vectors at each point
are calculated by considering each point as the origin. A graph

D. Calculation time
For dexterous manipulation, we need to obtain haptic
information from multiple sensors. Therefore, we must
process a large amount of information from different sensors
simultaneously. In this section, we evaluate the calculation
times for the finger-shaped haptic sensor and four sensors.
The calculation time is obtained from the frequency of theCPU. The speed of the processor used in the PC in thisexperiment is 2.8 GHz (Intel® Pentium®D processor). The
program runs on Windows XP SP2. The accuracy ofmeasurement is 1 µs. The calculation time is measured from100 trials, and the average and standard deviation of thecalculation time are shown in Table II.
In this prototype, the camera unit captures four images at 67
fps. Therefore, the PC must calculate the force vector fields ofthe four sensors within 16 ms. The result satisfies thisrequirement even when we process the information from thefour sensors simultaneously.

492them in order to realize the desired specifications.
The time resolution of the sensor can also be improved. In
the prototype sensor, the calculation time is less than 2.5 mseven after using four sensors. Therefore, a time resolution of
400 Hz can be realized by improving of the frame rate of thecamera.
The finger-shaped haptic sensor is capable of performing
basic functions and can be improved further. Therefore, we
believe that the implementation of this haptic sensor will
enable robotic hands to perform dexterous manipulations.

Normalized force f

1.00(5.0, 4.0)

0.75

(5.0, 4.5)
(5.0, 5.0)

0.50

(5.0, 5.5)
(5.0, 6.0)

0.25
0.00
-7.5

-5.0

-2.5

0.0

2.5

5.0

7.5

Distance from force applied point y [mm]V. CONCLUSION

Normalized force f

1.00(4.0, 0.0)

0.75

(4.5, 0.0)
(5.0, 0.0)

0.50

(5.5, 0.0)
(6.0, 0.0)

0.25
0.00
-7.5

-5.0

-2.5

0.0

2.5

5.0

7.5

Distance from force applied point x [mm]Fig. 9. Force measured by high-density sampling. The first and second graphs
show the results of the application of force at the center and the edge of thesensor, respectively.
TABLE IICALCULATION TIME
Number of sensors
1
4

Average [µs]
545
2159

In order to enable a robotic hand to perform dexterous
manipulations, we developed a finger-shaped haptic sensor
that can measure a three-dimensional force vector field. We
applied the technology used in the vision-based haptic sensor
developed by us to the finger-shaped haptic sensor. Because of
the complicated shape of the finger-shaped sensor, we used
actual measurements for the calibration in order to develop a
prototype of the sensor. We then evaluated the basic
performance of the sensor.
The results of the evaluation imply that further
improvement in the resolution of the sensor is possible. Hence,
we will further modify the finger-shaped sensor to enable a
robotic hand to manipulate objects and realize dexterous
manipulation by using the information on the force vector
field.
REFERENCES

Std.Deviation [µs]
[1]

56
43

[2]

IV. DISSCUSSION[3]

The finger-shaped haptic sensor exhibits linear response
and its force magnitude resolution is 30 gf in all the three
directions. Moreover, the spatial resolution of the sensor is 5.0
mm in the x- and y-directions at any point. These results showthat the finger-shaped haptic sensor can be used as a both force
vector sensor and a force field sensor.
The results also indicate a possibility for further
improvement in the performance of the sensor. The spatial
resolution of the sensor is identical to that obtained from thesampling theory. Therefore, the spatial resolution can be
improved by reducing the diameter of the markers and the
distance between them. For example, to obtain a spatial
resolution similar to that of the human fingertip(approximately 2.0mm [9]) for the sensor, the distance
between the markers and the diameter of each marker shouldbe 0.2 and 1 mm, respectively. In the prototype, the marker
diameter is 0.5 mm or 10-20 pixels, and the error in themeasurement of the marker movement is 0.15 pixel. Therefore,
we can reduce the size of the markers and the distance between

[4]

[5]
[6]
[7]
[8]
[9]

493

H.R. Nicholls, M. H. Lee, “A Survey of Robot Tactile Sensing
Technology,” Int. J. Robotics Research, Vol. 8, pp. 3-30, 1989.
M. H. Lee, H. R. Nicholls, “Tactile Sensing for Mechatronics – A Stateof the Art Survey,” Mechatronics, Vol. 9, pp. 1-31, 1999.
J. G. Silva, A. A. Carvalho, D. D. Silva, “A Strain Gauge Tactile Sensor
for Finger-mounted Applications,” on IEEE Trans. On Instruction andMeasurement, Vol. 51, No. 1, pp. 18-22. 1991.
M. Ohoka, Y. Mituya, K. Hattori, I. Higashioka, “Data Conversion
Capability of Optical Tactile Sensor Featuring an Array of Pyramidal
Projections,” IEEE International Conference on Multisensor Fusion andIntegration for Intelligent Systems, pp. 573-580, 1996.
N. J. Ferrier, R. W. Brockett, “Reconstructing the Shape of a Deformable
Membrane from ImageData,” The International Journal of Robotics
Research, Vol. 19, No. 9, pp. 795-816, 2000.
K. Kamiyama, H. Kajimoto, N. Kawakami, S.Tachi, “ Evaluation of a
Vision-based Tactile Sensor,” In Proc. of International Conference on
Robotics and Automation (ICRA2004), 2004, pp. 1542-1547.
L. D. Landau, E.M.Lifshitz, “Theory of Elasticity,”
ButterworthHeinemann, 1985.
W. Menke, “Geophysical Data Analysis: Discrete Inverse Theory,”
Academic Press Inc., 1989.
E. R. Kandel, J. H. Schwartz, T. M. Jessel, “Principles of Neural Science,
4th Edition,” McGraw-Hill, 2000.