GelForce: A Traction Field Tactile Sensor
for Rich Human-Computer Interaction
Kevin Vlack’
Hiroyuki Kajimoto4

Kazuto Kamiyama*
Naoki Kawakami’reasons for this, the first being the projected complexity
and cost of manufacturing a miniaturized, highperformance sensor that can measure force vectors withhigh density. Perhaps the second reason is that it is still
unclear what advantage this class of sensor would haveover established sensors such as force feedback devices
that measure a force vector at a localized point [4], orpiezoelectric element devices that measure densedistributions of force magnitudes [SI. We shall address
both concems in this paper by introducing a method tomeasure force applied to the surface of a silicone bodyusing computer-vision techniques, and presenting thenovel capabilities of this sensor through interactive
digital art.

ABSTRACT
Recent advances in vision-based tactile sensation havegiven rise to a novel class of high-performance sensing
devices that measure traction fields (i.e. distributions of
3-D force vectors) with density comparable to the
biological sense of touch. While this has been-an
emerging trend in robotics, it introduces diverse new
possibilities for human-computer interaction as well. We
describe how to apply computer vision techniques tomeasure the traction field applied to the surface of asilicone body, and discuss the potential of using thecomputed vector distribution as a rich and versatileinterface for interactive desktop applications.

2 VISION-BASED TACTILE SENSING

Keywords: vision-based tactile sensing, force vector
field user interface, vector field visualization

The proposed sensor has a relatively simple systemdesign, consisting of a transparent silicone elastic body,and a video camera placed undemeath the body tomeasure the intemal deformation according to force
applied from humans or other extemal sources (seeFigure I).

1 INTRODUCTION

The specific objectives for researchers in datavisualization and robotics are generally distinct, but notdivergent: they share the fundamental goal ofestablishing technology in which people can interact withmachines as if they are handling natural objects, in acyhemetic way that is truly human-oriented. Combining
their respective innovations would he of mutual benefit
and may also introduce new possibilities to the domainof human-computer interaction. In this spirit, we propose
a practical application of a tactile sensor that measures afield of force vectors as an intuitive computer interface
with a rich geometrical structure.
The distinctive feature of the tactile sense over theother senses is its integral relationship with our bodily
movement. We are generally unaware of this until welose sensation through anesthesia or loss of blood flow.
Simple tasks like grasping an egg become critically
impaired without the interplay between muscle
contraction and mechanoreceptors in the skin. In
particular, the weight of an object is felt not only bycontraction of the muscle hut also force parallel to theskin. Fragile objects are grasped safely by adjusting theproportion of perpendicular and parallel force. This
aspect has been studied extensively in the robotics
literature [1][2][3], and a variety of miniaturized tactile
sensors have been developed to endow mechanical handswith the capability to perform fine manipulation tasks.
However, to our knowledge there have been few or noeffons to introduce traction field sensors into the forumof interactive desktop applications. We suspect twoFigure 1: Desktop style vision-based force sensor composed ofrectangular transparent silicone body.
2.1 Derivation of Force from Elastic Displacement

2. I . I Elasticity Theory

Perhaps the most scrutinized problem ‘in elasticity theoryis the derivation of deformation of an elastic body given
an applied stress. If we assume the elastic medium islinear, isotropic, and homogeneous, the linear relationand strain
between three-dimensional stress E = b,,.,,. ,.

E,,.=I: 1(1)-“lu,+u:)]where E is Young’s modulus characterizing the stiffnessof the material, and v is Poisson’s Ratio characterizing
the compressibility of the material. This fundamental
relationship can reframed for a continuous elastic
medium as a convolution to relate the internal

miZOta~StaT.t.U-tOliYO.aclP

University of Tokyo, kaji~4srar.t.u-tokyo.acJp‘University of Tokyo. kawakami~rtar.t.u-tokyoacjp’University of Tokyo. i a c h i ~ ~ i a r . t . u - t o k y o . ~ ~ . j p0-7803-8564-0/04/$20.0 02004 IEEE) is expressed through Hooke’s law [6] as:
G , = qEu . - v ~ u , + u : ) ]’ University of Tokyo. vlack~ilsrar.i.u-tokyo.ac.jp’ University of Tokyo. kazuto~~lar.r.u-tokyo.acJp‘ CASTI.

Terukazu Mizota’
Susumu Tach?

-11-

deformation of the elastic body to the distribution of
external force applied at the surface, also known as atraction field. Expressing this convolution in matrix form,
we may derive force from deformation by solving theinverse problem. Please refer to [7] for details.

The developed sensor has proven to he exceptionallyadept at measuring qualitative information unobtainable
by existing sensors, so it is a rather promising tool for ahost of applications in visualization, as well as tract
analysis applications in sports, industry, and medicine. It
can function as an intuitive computer interface like thetouchpad or mouse, except with much more stmcture.
This characteristic provides lush input for real-time
physical simulations. It also makes it a powerful means
to interact with virtual environments modeled withphysical parameters. We envision accompanying thetactile sensor with more conventional input devices suchas a mouse or trackball in order to adaptively apply realtime traction field input to familiar interactive
applications such as image processing, CAD design,
image processing, sound synthesis, and computer
entertainment (Figure 2).
It is no exaggeration to say that every scientifictechnology is based on sensors. For example, audio
recording cannot exist without the measurement of sound,and television broadcasts would be impossible without
catching radio waves as picture signals. Just as the video
camera and microphone adeptly measure information forvisual and acoustic sensation, we believe the coretechnology of the developed sensor has great potential todevelop mainstream devices to enhance our daily lives.
The first reason is that the sensor can measure complete
information about force, that is direction, magnitude anddistribution. with the same versatility as the humantactile sense. Second, its sensing mechanism is simple, so
the sensor can adapt various shapes and be manufactured
cheaply. Finally, powerful and intuitive input devices
will surely he needed to accommodate emerging markets
related to continual improvements in processing power.

2.2 Measuring Elastic Deformation
There are a myriad of techniques to measure visualdisplacement [8],with a general tradeoff between speed,accuracy and robustness. Fortunately, tracking thedeformation of a stiff, energy-conservative silicone body
is a highly specific problem with considerable
geometrical constraints, which may he exploited todevelop a specialized method with both high speed andaccuracy. The proposed sensor uses a CCD camera
placed beneath the transparent elastic body to track
interior points distinctly indicated by circular markers.
Using a center-of-mass calculation, a displacement
distribution of the marken is tracked to suh-pixel
accuracy, with a precision of approximately 40 pm.
Although a single camera can only obtain 2D
information, by aligning different colored markers at
separate depths within the silicone body, we obtain 3D
information by their corresponding differences indisplacement resulting from the applied force. Through alinear least squares method, the traction field may hecomputed quickly with a matrix-vector multiplication
using the precomputed convolution matrix.

3 APPLICATIONS, PRESENT AND FUTURE
A major design criterion for this sensor is to serve asan affordable traction field input device for commodity
PC systems. It is critical to achieve a high density ofsensing to appeal to the human senses, and with a highrefresh rate to realize the concept of "What-You-See-IsWhat-You-Touch.

Patent Information: Application Number (2000)264407.
(2004)177786. (2003)322624.(2003)322627, (2004)i77794
REFERENCES[ I ] R. S. Feanng and 1. M. Hollerbach. "Basic Solid Mechanics for
Tactile Sensing". lntemational Journal of Robotics Research Vol.
4. No. 3. 19x5. pp. 40-54[2] N . J. Ferrier. and R. W. Brocketl. "Recansrructmg the Shape o f a
Deformable Membrane from Image Dara". InIemamnal JoumalofRobotics Research Vol. 19. No. 9.2000, pp. 795-816.

[3] E. J. Nicholson. and N. J. Fenier, "The Reliability o f CurvaNre

[4]
[5]

Estimates from Linear Elastic tactile Sensors". lniemaiional
Conference on Robotics and Automation 1995. pp. 1126-1 133
M. Sato. Y. Hiram and H. Kawarada. "Space interface device for
artificial realiry---SPIDAR: System Computing Japan (USA).
vol. 23. No. 12. 1 9 9 2 . p ~44--54.
.
D.T. V. Pawluk. 1. S. Son. P. S. Wellman, W.1. Peine. R. D,Sensor for Biomechanical
Biomechanical Engineering.
Transactions of the ASME. Vol.lZ0. No.2. 1998. pp. 302-305.

Howe.

"DistributedMeasurements".

Pressure

Joumalof[6] S . Timonshenko and 1. N. Goodier. Theory o f elasticity New
York: McGraw-Hill. 1951

[7] K. Kamiyama. H. Ka~imolo. N. Kawakami. and S . Tachi.
"Evaluation of a Vision-based Taciile Sensor". IEEE lntemaiional'Conference on Robotics and Automation. 2004. (submitted)[8] 1. Yha. "Estimation of 2D displacement field based on aftine
geometric invariance and scene constraints". lntematianal Joumsl
ofcomputer Vision Volume 46. No. I, 2002. pp. 25-50.

Figure 2: Applying a force vector field tactile sensor loachieve sophisticated visual effects-12-