US007659502B2(12) Ulllted States Patent
Tachi et a].

(10) Patent N0.:

US 7,659,502 B2(45) Date of Patent:(54)

OPTICAL TACTILE SENSOR

(56)

(75)

Inventors: Susumu Tachi, Tsukuba (JP); Naoki

Feb. 9, 2010

References Cited
U_S_ PATENT DOCUMENTS

KaWakami, Tokyo (JP); Hiroyuki
Kajimoto Yokohama (JP). Kazuto-’-

KamIyama’MaebaShIUP)’Terukaz“Mizota, Yamatokohriyama (JP)(73) Assignee: Toudai TLO, Ltd., Tokyo (JP)*N ot1ce:'S u bj ect to anyd'1sc l a1mer,'t h e term 0 f t hi spatent is extended or adjusted under 35(21) Appl. No.:
(22)
(86)

U.S.C. 154(b) by 440 days.
10/571,370

PCT Filed:
PCT No.:4,599,908 A *’.

Mar. 3, 2005
PCT/JP2005/003650

6,909,084 B2

2003/0178556 A1*

7/1986 Sheridan e161. ...... .. 73/862046
6/2005

Tachietal.

9/2003

Tachietal. .......... .. 250/22711............... .. 250/221

FOREIGN PATENT DOCUMENTS
JP
JP
JP
JP
WO

7-67020
11-108630
11-118625
2001-141660
02/18893 Al

3/1995
4/19994/19995/20013/2002* cited by examinerPrimary ExamineriGeorgia Y Epps
§ 371 (0)0),(2), (4) Date:(87)

Assistant ExamineriDon Williams

Mar. 9, 2006

PCT Pub. No.: WO2005/124305(57)

PCT Pub. Date: Dec. 29, 2005

An optical tactile sensor and an image information integrat(65)

Prior Publication Data
US 2008/0245955 A1

(30)

(52)
(58)

Oct. 9, 2008

Foreign Application Priority Data

ABSTRACT

ing method capable of measuring large area force vector
distribution are provided. The optical tactile sensor comprisesa tactile section comprising a transparent elastic body (1) anda plurality of markers (3, 4) provided therein, and a plurality

G06M 7/00(2006.01)

of imaging devices (6, 6) for obtaining a marker image by
photographing the behavior of the markers When an object (5)contacts the tactile face (2) of the transparent elastic body (1).
The method comprises a step of obtaining a partial image by
photographing the partial regions A, B, C and D of the trans

H01L 27/00
G06F 3/041(2006.01)(2006.01)

parent resilient body (1) using the plurality of imaging
devices (6, 6) such that each imaging device (6) has an over

US. Cl. ................... .. 250/221; 250/208.1; 345/173
Field of Classi?cation Search ............... .. 250/221,

lapped photograph region (11), and a step of integrating thepartial images obtained by each imaging device (6) such thatthe identical markers in the overlapped photograph region

Jun. 16, 2004

(51)(74) Attorney, Agent, or FirmiLadas & Parry LLP(JP)........................... .. 2004-177786

Int. Cl.

250/231.1, 216, 226, 208.1, 231.19, 559.44,
250/559.46; 345/173, 175; 356/601, 607;
73/774, 768; 901/33
See application ?le for complete search history.

match.

17 Claims, 11 Drawing Sheets

U.S. PatentFeb. 9, 2010

FIGJ

0

i

Sheet 1 0f 11

US 7,659,502 B2

US. Patent

FIG.3

Feb. 9, 2010

Sheet 2 0f 11

US 7,659,502 B2

US. PatentFeb. 9, 2010

Sheet 3 0f 11

US 7,659,502 B2

FIGS
IMAGING REGION OF CAMERA I

12IMAGING REGION OF CAMERA 2._,_ _. _0 BLUE MARKER'0 RED MARKER

l

I

l

l

I

|

I

(‘I
l

I

\I. !\

Il—-*_In

I I I

I

I

l

l

I

I

I

I

l

I

l

US. PatentFeb. 9, 2010

Sheet 4 0f 11

US 7,659,502 B2

FIG.7
<>

z

<2

011» "<6
X

(a)

" +0

0+

M2.

.1001)‘

ffm \f<2>

“(3)

1

s

FIGS

2

a),
4

US. Patent

FIG.9

Feb. 9, 2010

Sheet 5 0f 11

US 7,659,502 B2

US. PatentFeb. 9, 2010

US 7,659,502 B2

Sheet 6 0f 11

FIG.10'Q 3. ‘=2. ‘ O ’‘ 5.. 1. "1.5.11. D O “O

.00 0.0.00 00 0- 0. o 0 Q00 0;

90950390969969.5900

m..

o: 0 105,000 io-.-<-o:fo: o. _ 0 010-0; 0
.|_-

n

-

u

n

u

-

-.-Z -

_

n

-

a

-

I

_

--_--.-|-__"i;-l". B LUE MARKER

A Point for Obtaining Applied Force"""" Neighboring Region

0 RED MARKER

US. PatentFeb. 9, 2010

Sheet 7 0f 11

US 7,659,502 B2

FIG.11

O

0

O
O
O
O
O
0
0000000

00000000
0000.00
00000000000
0000000
OOOOOOOOOOOOOO
O

O

O

O
O

\

O

O

(3

O
O

. Q I Force Sampling Points. ; Computational Results used
after computationjUS. PatentFeb. 9, 2010

Sheet 8 0f 11

US 7,659,502 B2

FIG.12
Fl F2

III-I

‘Y

mxn'nmsm4""'"mzs.OIOIOO'IOIIOOO'DIIOIIIX

njJ?kI Movement Vector of Observed Marker

Fi: Force Vector

FIG.13
E! E2...“''

m1m2m3m4'"'"'

m23....................'..
$®$Q$$$®§®Q$???$
IllIl2ll3Il4"""'1122

X

nJJnkI Movement Vector of Observed Marker

F i: Force Vector

US. PatentFeb. 9, 2010

Sheet 9 0f 11

US 7,659,502 B2

FIG.14
Pl! .151...“
-

0

-

.

-‘.'_2- I Marker (Movement information is not used)X

IljJIlkI Movement Vector 0f Observed Marker
7‘ F12 Force Vector

if;- I Marker (Movement information is not used)xmm: Movement Vector of Observed Marker
Fi: Force Vector

US. PatentFeb. 9, 2010

Sheet 10 0f 11

US 7,659,502 B2

F IG.16

WHH IIIHIHH |\\|~\ #0"'“Hmllrlllml

FIG.17

300

1,100

30

US. PatentFeb. 9, 2010

FIG.18FIG.19

Sheet 11 0111

US 7,659,502 B2

US 7,659,502 B2
1

2

OPTICAL TACTILE SENSOR

Patent Reference: WO02/l 88923 Al

FIELD OF THE INVENTION

tactile sensor capable of measuring force distribution over alarge area.

An object of the present invention is to provide an optical
Another object of the present invention is to provide amarker information acquisition method and a marker image

The present invention relates to an optical tactile sensor,
and preferably to a tactile sensor used for obtaining forces
applied to a relatively larger area.

integration method capable of measuring force distribution
over a large area. Still another object of the present invention
is to provide a force vector reconstruction method employedin an optical tactile sensor capable of measuring force distri

BACKGROUND OF THE INVENTION

bution over a large area.

When considering understanding the contact state of a
SUMMARY OF INVENTION

contact surface using a tactile sensor, there are vectors of

three components representing magnitude and direction of
The present invention has been conceived in order to solve

force acting at each point of the contact surface. This isrepresented as f(x,y) in the coordinate system of FIG. 1. Here,these problems. According to the present invention, there is

f is a vector, and so actually has three components x, y and Z

provided an optical tactile sensor comprising a tactile section

having a transparent elastic body and a plurality of markers
provided therein, and imaging means for obtaining a marker

at each point. When explicitly expressing each component, it
is represented as f(x,y):[fx(x,y), fy(x,y), fZ(x,y)]. Since force

distribution has three components at each contact point, in 20 image by photographing behavior of markers When an object

contacts the sensing surface of the transparent elastic body.
The optical tactile sensor is characterized in that the imaging
means comprises a plurality of imaging devices, and each of
the plurality of imaging devices is provided such that each oforder to reconstruct force distribution for each contact surface

using a tactile sensor, it is necessary to acquire informationfor each contact point on the contact surface With at least threedegrees of freedom.

the imaging regions Which is captured using each of theSome of inventors of the present invention et al. have 25 respective imaging devices has an overlapped imaging region

proposed an optical tactile sensor that is capable of measuring
three-dimensional force vector distribution. A principle of the

Which partially overlaps; the optical tactile sensor further
comprising image integration means for integrating each of
the marker images Which are obtained using the plurality of

optical tactile sensor Will be explained based on FIG. 2. The

optical tactile sensor comprises a transparent elastic body andimaging devices, and the image integration means integrates
a CCD camera. By photographing spherical markers embed
the imaging regions such that identical markers in the over30
ded in the transparent elastic body by the CCD camera, interlapped imaging regions are matched, to thereby form an inte
nal strain information of the elastic body is measured When agrated image.
force is applied on the surface of the elastic body, and force
According to one preferred aspect, the tactile section com
vector distribution is reconstructed from the information.
prises a large area tactile section Which is formed by combin
By taking an image of the spherical markers by a CCD
ing a plurality of small area tactile sections. It should be noted
camera from Z-direction Where an elastic body surface is 35 that regarding the relative expressions such as a “large area”
taken as the x-y plane and an orthogonal direction to the x-yplane is taken as the Z-axis, movement of a point to be mea
sured When force is applied is measured as a movement vector
in the x-y plane. To reconstruct the force vector distribution

and a “small area”, in this speci?cation, a “large area” means
larger compared to a tactile section such as is to be incorpo
rated into a robotic hand, While a “small area” means smaller

compared to a “large area”. While a seating surface of a chair,from the strain information, N><N red spherical markers and 40 a bed surface, a ?oor surface, and so forth may be listed asblue spherical markers are arranged at different depths in theexamples of a “large area”, objects that are smaller than theseelastic body as points to be measured to obtain tWo sets of
tWo-dimensional movement vectors With different depths asbution.
As a use for such an optical tactile sensor, initially, appli

items are also included in What is referred to by a “large area”.
Further preferably, the tactile sensor comprises a unit com
prised of one small area tactile section and one imaging
means corresponding to the small area tactile section. The
tactile sensor is formed by combining a plurality of the units.
Formation of the tactile sensor using sensor units enables

cation to a robotic hand of a humanoid robot and so forth iscreation of a tactile section having a desired area.

tWo pieces of different information, thereby increasing theamount of information to reconstruct the force vector distri

considered, and as an application for an optical tactile sensor,
study has focused on a small built-in type sensor. HoWever,such an optical tactile sensor, Which is capable of measuring
three dimensional force vector distribution and has a sensor

45

50

The present invention employs another technical means
including a method for obtaining a marker image using an
optical tactile sensor comprising a tactile section having atransparent elastic body and a plurality of markers provided

surface made of a elastic body, is expected to ?nd application

therein, and an imaging means for obtaining a marker image
in a variety of ?elds, not only to a robotic hand.
by photographing behavior of a marker When an object con
As one of the expected applications, use of an optical 55 tacts the sensing surface of the transparent elastic body. The
tactile sensor as so called an environmental type tactile sensor

can be considered. In this speci?cation, as compared to a
built-in type tactile sensor Which is incorporated for use into
a robotic hand or the like, a tactile sensor Which is used in a?xed manner in an environment is referred to as an environ

method of obtaining marker information comprises a step ofproviding a plurality of imaging devices as the imaging
means and obtaining a partial marker image by photographing a partial region of the transparent elastic body using eachof the plurality of imaging devices such that each imaging

mental type tactile sensor. HoWever, With respect to an envi 60 device has an overlapped imaging region, and a step of form
ing an integrated marker image by integrating the partialronmental type tactile sensor, Which is used in a ?xed mannerin an environment, measurement of force distribution over alarge area is expected to be necessary because such a sensor ismarker images obtained using the imaging means such that
identical markers in the overlapped photograph regions arematched.
assumed to be applied to, for example, a seating-surface of a
The present invention employs still another technicalchair, a bed, a ?oor, or the like. This hinders application of a 65
means including a method for integrating a marker image,conventional small optical tactile sensor, Which is assumed to
be incorporated in a robot hand or the like
Which is employed in an optical tactile sensor comprising a

US 7,659,502 B2
3

4

tactile section having a transparent elastic body and a plural

FIG. 7 is a diagram explaining a force vector applied to aity of markers provided therein, and a plurality of imaging
devices for obtaining a marker image by photographing

contact surface (plane surface) and movement of the marker.

behavior of a marker When an object contacts the sensing

contact surface (free curving surface) and movement of thesurface of the transparent elastic body. The image integration
method comprises a step of obtaining a partial image by
photographing a partial region of the transparent elastic bodyusing each of the plurality of imaging devices such that each
imaging device has an overlapped imaging region, and a step

marker.
FIG. 9 is a diagram explaining a method for creating aof integrating the partial images obtained using the imaging

using a plurality of cameras and explaining a method for
reducing the time for calculation necessary for reconstructionof a force vector, in Which, When a force applied to a point isobtained, data on only the markers located in its vicinity areused in the calculate of the force.

FIG. 8 is a diagram explaining a force vector applied to atransfer function for use in reconstruction of force vector

distribution.

FIG. 10 is a diagram shoWing an integrated image capturedmeans such that identical markers in the overlapped photograph regions are matched.
As a construction of the tactile portion, preferably, thetactile section comprises a transparent elastic body and aplurality of groups of markers provided inside the elastic
body, each marker group being made up of a number ofcolored markers, With markers making up different marker
groups having different colors for each group. The imaging
device takes an image of the behavior of colored makers in thetransparent elastic body When an object contacts the surface

FIG. 11 is a conceptual diagram explaining an improved
manner of the calculation reduction method. In the draWing,

black and Whit circles represent sampling points for force,

imaging means While using a sensing marker. As the present

and the black circle shoWs a result of calculation to be used
after the calculation.
FIG. 12 is a diagram shoWing an increased number ofmarkers.
FIG. 13 is a diagram shoWing an increased number ofmarkers, in Which movement of a marker caused relative toinvention comprises a plurality of imaging devices, calibra

the force applied to one point is focused.

20of elastic body.
Also, the present invention includes calibration of each

FIG. 14 is a diagram explaining the speed increasingtion of the imaging means is required. As the present invention has a sensing marker as an essential component, the 25 method shoWn in FIG. 10, based on FIG. 13.
FIG. 15 is a draWing explaining an improved method,marker can also be used as a calibration marker. Further, thebased on FIG. 13.
present invention also includes a method for reconstructing aFIG. 16 is a diagram shoWing another embodiment of aforce vector While using such an optical tactile sensor. Whena force vector is calculated Without the data on the behavior ofmarker (cylindrical marker).
FIG. 17 is a diagram shoWing another embodiment of aa marker With less contribution to the force vector calculation, 30
time for force vector calculation can be reduced.
marker. The upper diagram shoWs a stepWise band marker,

According to the present invention, as the information on
the behavior of a marker is obtained using a plurality ofWhile the loWer diagram shoWs a pyramidal marker.

imaging means, even a tactile section having a large area canmarker (crossing strip marker).

FIG. 18 is a diagram shoWing another embodiment of abe preferably handled. Therefore, use of a sensor according to 35
the present invention makes it possible to measure force vec
tor distribution over a large area, Which in turn makes it
possible to determine a force applied to a sensing surface
having a large area, based on the obtained marker informa
tion. Moreover, as a sensing marker can be used for integration of the image data obtained by the imaging means and 40

FIG. 19 is a diagram shoWing another embodiment of amarker (color-discriminated plane marker).
DETAILED DESCRIPTION OF THE PREFERREDEMBODIMENT[A] Basic Construction of Optical Tactile Sensor

calibration of the respective imaging means, the sensor can beformed using a reduced number of components.

An optical tactile sensor of the present invention comprises

BRIEF DESCRIPTION OF THE DRAWINGSa sensing section and imaging means. The tactile section
comprises a transparent elastic body and a plurality of marker
45

FIG. 1 is a draWing shoWing force vector distributiongroups provided in said body, each marker group being com
prised of a number of colored markers, With markers consti

tuting different marker groups having different colors foreach group. The imaging means is provided to take an image
FIG. 2 is a draWing shoWing the principle of an opticalof behavior of colored markers When the surface of elastic
tactile sensor. The upper diagram is a plane vieW (CCD
image) of a transparent elastic body, While the loWer diagram 50 body is contacted by an object to obtain marker images. A
force applied to the surface is reconstructed from information
is a side vieW of the transparent elastic body. The transparent
as to the behavior of markers that is obtained from the marker
elastic body has tWo kinds of marker groups embedded
images.
therein. When force is applied to the transparent elastic body
At least one of displacement, strain and inclination of thefrom beneath, the marker moves from What is shoWn in thecolored markers When the elastic body contacts an object is
55
left diagram to What is shoWn in the right diagram.
observed by photographing behavior of the colored markers.
FIG. 3 is a schematic diagram shoWing an optical tactile
Strain information inside the transparent elastic body is
sensor according to the present invention.
detected from information about the behavior colored mark
FIG. 4 is a schematic diagram explaining enlargement of
ers When a contact object contacts the sensor, and the shape of
the area of a sensor surface. The left diagram shoWs one unit
the contact object calculated from strain information, andexerted betWeen a tactile sensor and an object to be contacted.

comprising one CCD camera and an elastic body section,

While the right diagram shoWs the result of area enlargement60

information about force acting on a contact interface (includ

ing both the elastic body surface and the contact object surface) are also detected. According to the present invention, it
FIG. 5 is a diagram for shoWing a method for integrating
is possible to separately collect a plurality of types of infor
the image data Which is obtained using a plurality of imaging
mation With a simple method called “color coding”, and it ismeans.
65 possible to acquire a plurality of types of tactile information
FIG. 6 is a schematic diagram shoWing image data Which isat the same time With an optical system. According to the

combining units.

obtained using the sensor shoWn in FIG. 4.

present invention, independent observed information (infor

US 7,659,502 B2
5

6

mation as to behavior of markers) Whose number is equal to or
greater than the number of unknowns are collected using

When an object 5 comes into contact With the transparent

color coding, and it is possible to estimate and reconstructforce vectors by stably resolving an inverse problem.
The colored markers are photographed by photographingdevice, in a preferred example, a CCD camera, and image
processing is carried out by a processor. For example, animage at the time of body contact and an image of a previous
condition (a condition Where external force is not acting on
the transparent elastic body) are compared, and an amount of
movement of the markers is detected. Alternatively, the markers are embedded in the transparent elastic body in such anarrangement that they can not be recogniZed normally (in astate Where external force is not acting on the transparent
elastic body), and a con?guration is such that markers arerecogniZed in response to displacement deformation andinclination of markers caused by strain in the vicinity ofpositions Where each of the markers exist When an object
contacts the transparent elastic body, and information iselastic body 1, the colored markers 3, 4 provided inside thetransparent elastic body 1 are moved due to the internal strainof the elastic body. The sensor is also provided With a camera
6 as a photographing device and a light source 7. The opticalcamera 6 is arranged at a position on an opposite side to Wherean object 5 contacts so that the transparent elastic body 1 isprovided betWeen the optical camera 6 and the object 5, andbehavior or movement of the markers 3, 4 is photographed by
the camera 6. The light source 7 may transmit light through a

Waveguide such as an optical ?ber for example. Images ofmarkers 3, 4 obtained by camera 6 as imaging means aretransmitted to a computer 8 and the marker images are dis

played on a display. The processor of the computer 8 calcu
lates the marker information (movement vectors as move

ment information, for example) regarding the behavior(displacement, strain or inclination) of markers. The proces
sor reconstructs the distribution of forces applied to the sur

face 2 by an object 5 using the marker information (movement
information, for example) and a transfer function that is

detected from the appearance of the colored markers. In 20 stored in the memory device of the computer 8.

another preferable aspect, the behavior of markers (step-like
strip markers, for example) can be obtained by variance ofmarker intensity.

The transparent elastic body 1 is preferably made of silicone rubber, but it can also be made from another elastic
material such as another type of rubber or elastomer. The
markers are preferably made from an elastic material, andmore preferably made from the same material as the trans

The optical tactile sensor stores a transfer function by
Which force vectors or force vector distribution applied to the 25
surface of the elastic body are reconstructed from information
parent elastic body 1. In one preferred embodiment, the col(movement vectors of each marker When an object contacts
ored markers are formed by adding pigment to silicone rub

the surface, for example) obtained by photographing device

ber. Since deformation of the elastic body should not be
inhibited by the markers, the markers are also preferablythat associates force information applied to the surface of themade from an elastic material (preferably having the same
30
sensor With information as to the behavior of markers (moveelastic constant as the elastic body). The material of the markment vectors, for example). The image information of markers is not particularly limited as long as the extent to Whichers is obtained by photographing the colored markers Whendeformation of the elastic body is inhibited is suf?ciently
the object contacts the sensing surface of the elastic body, andsmall. It is also possible for a part of the elastic body tothe information as to the behavior of markers is obtained fromconstitute the markers.
the image information of markers. The force vector is 35
With the present invention, a plurality of optical markers
obtained as an output by inputting the obtained information toare distributed Within the transparent elastic body 1, andthe transfer function. The number of information as to theinformation about a contacting object and information about
behavior of markers that is input to the transfer function isdisplacement and deformation Within the elastic body pro
more than the number of force vectors to be obtained.
duced by contact are detected by photographing situations
The transfer function, depending on the shape of the elastic 40 Where displacement, deformation and inclination arise in thebody, may be obtained based on an equation derived frommarkers due to deformation of the elastic body 1 as a result of
theory of elasticity. HoWever, When the surface of elastic bodythe object coming into contact With the elastic body 1 using a
is an arbitrary curved surface, preferably, the transfer function
camera. FIG. 3 shoWs tWo marker groups, but the number ofis obtained by measurement or simulation. The transfer func
marker group is not limited, and three marker groups may betion by measurement or simulation can be obtained from 45 located in a layered manner along the surface 2.
information (movement vectors, for example) as to behavior
A camera, as a photographing device, is a digital camera,of markers When x-directional force, y-directional force, andnamely a camera for outputting image data as electrical sig
Z-directional force having predetermined magnitude, fornals, and in one preferred example is a CCD camera. It is also
example, are applied to sampling points arranged on the surpossible to use, for example, a digital camera using a C-MOS
face of the sensor.
50 type image sensor. If three types of markers are prepared in
Referring to FIG. 3, the construction of an optical tactile
red, green and blue, there are tWo methods of perceiving thesesensor of the present invention is shoWn. The sensor com
three colors individually. The ?rst method is to use color
prises a transparent elastic body 1 formed of a transparent?lters for separation Where each marker can be regarded asas to behavior of markers. The transfer function is a function

elastic material and a curved surface 2, or a surface for sens

being individually photographed directly by looking at RGB

ing. The transparent elastic body 1 is provided With a plurality

output from the camera. The second method is a methodof colored markers 3, 4 embedded in the transparent elastic
body 1 in the vicinity of the surface 2 and along the curved
surface 2. A sensing section is comprised of the transparent
elastic body 1 and the colored markers 3, 4 arranged inside theWhere imaging elements perceive only light intensity andelastic body.

light sources of red green and blue are prepared. When red isshone, light is only re?ected from the red markers While thered light is absorbed by the markers of the other tWo colors,and so the camera effectively only perceives the red markers.

The colored markers 3, 4 are comprised of tWo groups of 60 If this is also carried out at separate times for green and blue,
information equivalent to that using the ?rst method can becolored markers and the tWo marker groups are embedded indifferent depths respectively from the surface 2. Coloredacquired.

markers 3 constituting one marker group and colored markers[B] Environment Type Tactile Sensor
4 constituting the other marker group have different colors toeach other. For example, one marker group consists of a 65
An embodiment of an environmental type tactile sensor
plurality of blue markers 3 and the other marker group con
sists of a plurality of red markers 4.
according to the present invention Will be described. An envi

US 7,659,502 B2
7

8

ronmental type tactile sensor comprises a plurality of sensor
units. As shoWn in the left diagram in FIG. 4, a sensor unit

determination of the position and orientation of the camera in
the World coordination system. Occurrence of distortion due
to a lens is inevitable When a Wide lens is used. The positionand orientation of the camera is most accurately determined
When using image data. In this tactile sensor, as correlation
betWeen the image data and the actual position needs to be
determined, it is necessary to apply camera calibration. Gen
erally, in a measurement system using a plurality of cameras,

comprises one small area tactile section 10 and one CCD
camera 6 Which serves as an imaging means for photographing the small area tactile section 10. As described above inconnection With a basic structure, the small area tactile section 10 comprises a transparent elastic body and coloredmarkers provided inside the transparent elastic body. Whenan object contacts the sensing surface of the transparent elastic body, the colored markers provided inside the transparent
elastic body move, and the CCD camera 6 photographs the
movements of the colored markers. Then, as shoWn in theright diagram in FIG. 4, a plurality of sensor units are com
bined to form a sensor surface having a large area. By carpet
ing the small area tactile sections 10 so as to form the sameplane such that the edges of the tactile sections abut to one
another, a large area tactile section 100 is formed. The small

the folloWing procedure needs to be folloWed. Initially, a
stripe or black-White tile pattern With knoWn pattern intervals
is placed in a position Which is knoWn relative to the World

coordination system, and the pattern is photographed. Then,displacement from an image Which Would be captured Whenthe lens had no distortion is calculated using the captured
image, and the lens distortion and the position and orientationof the camera are determined. Here, in an environmental typetactile sensor Which similarly employs a plurality of cameras,
area tactile section 10 shoWn has a square shape in a plane
the stripe or black-White tile pattern With knoWn pattern intervieW. Although the shape of the small area tactile section isvals is already arranged in a knoWn position as a coloredmarker for sensing and ready to be acquired as an image.
not limited to square, a square tactile section is advantageous
When a plurality of small area tactile sections are carpeted. 20 Therefore, camera calibration, Which originally needs to be
conducted before assembly of the measurement system canAlso, although a small area tactile section having a planesensing surface 10 is shoWn in the draWing, the sensing sur
be conducted anytime after the assemblage.
face 10 is not limited to plane. A sensing surface having an

arbitrary curved surface is also applicable.
As a plurality of CCD cameras 6 are used, integration of the 25
image data obtained using the respective cameras 6 is neces[C] Method of Reconstructing Force Vector
Distribution on Sensing Surface

To obtain force vector distribution applied to a surface ofsary. FIG. 5 is a diagram explaining integration of the images
the sensor from obtained information (movement vectors ofobtained using a plurality of cameras. Such an image is dismarkers, for example) as to behavior of markers by an opticalplayed, for example, on a display of a computer 8. Initially,using a plurality of cameras 6, images of the small area tactile 30 tactile sensor, a transformation from information (movement
information, for example) M as to the behavior of markers tosection 10 are taken such that the respective photographforce information F is required. The transformation from theregions partially overlap to one another. Thereafter, themarker information M to the force information F is obtained
respective images from the cameras are integrated such thatby an equation FIHM. Referring to FIG. 7 and FIG. 8, a
the markers in the overlapped photograph regions 11 aremethod of reconstructing the force vector distribution frommatched, thus integrating image data. In FIG. 5, a black circle
represents a blue marker, While a White circle represents a red 35 the marker information Will noW be described based on amethod of obtaining the force vector distribution from themarker, and the photograph regions of the camera 1 and the
movement vectors of markers. FIG. 7 and FIG. 8 are substan
camera 2 have an overlapped photograph region 11. The blue
tially the same except that FIG. 7 shoWs a plane sensing
and red markers Within a region corresponding to the oversurface While FIG. 8 shoWs an arbitrary curved sensing surlapped photograph region 1 1 in the photograph region for thecamera 1 and those Within a region corresponding to the 40 face. Here, though, for the purpose of simpli?cation, only
tWo-dimensional section (y-axial direction is omitted) is con
overlapped photograph region 1 1 in the photograph region forsidered, an algorithm is the same for a general three-dimen
the camera 2 are matched to one another, Whereby the partial
sional space.
images captured using the camera 1 and 2 respectively areintegrated. It should be noted that although a spherical marker

Reference f refers to a force vector applied to a contact

is referred to here, a marker in other shapes such as is 45 surface, and references m and n refer to a movement vector ofdescribed later can be similarly used for image integration.
An image captured using four CCD cameras shoWn in FIG.
4 is shoWn in FIG. 6. Supposing that the respective photoa blue marker and movement vector of a red marker in the

CCD element. Discrete ?nite points (four points in FIG. 7 and

FIG. 8) are considered. As foregoing, force vector distribution
has three components (x component, y component and Z
are referred to as A, B, C, and D, the photograph regions A, B, 50 component), but only tWo components (x component and Z
component) are considered. Generally, taking an image by aC, and D are integrated such that overlapped photographcamera means a projection of a three-dimensional object to aregions 11 are resulted in Regions A and B, Regions A and C,

graph regions photographed by the respective CCD cameras

pixel plane of a tWo-dimensional plane so that marker movements only in the horiZontal direction (x component and yshould be noted that markers are omitted from FIG. 6. Eachcomponent) are projected in the plane. Here, marker movecamera is con?gured so as to photograph the behavior of the
55 ment only in x direction component is observed.
markers located in a region allocated in advance to the cam

Regions B and D, and Regions C and D, respectively. It

Here, eight components, f:[fx(1), fx(2), fx(3), fx(4), fZ(1),era, and the respective allocated regions have mutually overlapped regions. According to one aspect, each of the respec

fZ(2), fZ(3), fZ(4)] are force vector distribution to be obtained,

tive cameras is set in advance so as to photograph awhere m:[m(1), 111(2). m6). 111(4)] and PM). 11(2), no).

predetermined partial region of the transparent elastic body,n(4)] are movement vectors to be measured. The vectors mand con?gured such that integration of the images of the 60 and n are represented as X. Namely, X:[m(1), m(2), m(3),respective predetermined partial regions obtained using them(4), n(1), n(2), n(3), n(4)]. Here, movement vectors m and n
respective cameras enables formation of the entire image ofthe transparent elastic body.
Calibration of a camera to be applied When a plurality ofthat are observed When a unit force (magnitude of l) in thex-direction is applied to a point 1 are represented as Mx(1).

Namely, MX(1):[II1(1), 111(2), 111(3), 111(4), I1(1), I1(2), 11(3),imaging means (CCD cameras) are used is described. Gen 65 n(4)] When f:[l, 0, 0, 0, 0, 0, 0, 0].
Similarly, a movement vector of each marker When a unit
erally, camera calibration is applied for correction of distor
tion caused in the captured image due to the lens and also forforce in the Z-direction is applied to a point 1 are represented

US 7,659,502 B2
10
data up to eight by observing a movement of each marker inthe tWo layered marker groups.
In case of three-dimensional space (Where y-axis is added
to the draWing), at a point, a force vector has three degrees ofas MZ(1), a movement vector of each marker When a unit

force in the x-direction is applied to a point 2 are represented
as Mx(2), and so on. In case of a linear elastic body Where

linear summation relationship holds betWeen applied forces
and strains (most elastic bodies meet this characteristics),

freedom, and a horiZontal movement vector of markers has

tWo degrees of freedom. If the number of sampling points isfour, the number of unknowns f is tWelve,

movement vectors are represented as

Where f:[fx(1), fy(1), fZ(1), fx(2), fy(2), fZ(2), fx(3), fy(3),fZ(3), fx(4), fy(4), fZ(4)], Whereas the number of observed

When general forces f:[fx(1),fx(2),fx(3),fx(4),fZ(1),fZ(2),fZ(3),fZ(4)] are given. Conversely, the fact that the movement 10 movement Vectors is eight and is insu?iciem,
vectors can be re resented as fore oinmeans that su eo-isition of forces hpolds, therefore, tTie elgastic body is aplirlirear

elastic body‘

_

_

_

(1)’ myu)’ mx(2)’ mya)’ mx(3)’ my(3)’By providing tWo layered markers, it is possible to obtain

When the equat1on1s represented as a matrix form, X:H><f,.

.

Where H:[MX(1); MXQ); . ~ ~ ; MZ(4)]' The H is Called a 15 s1xteen~observed databy observ1ngthe layeredmarkers andto
transfer function because the H is a map that transfers a force
fto deformation X_ The matrix form Written with an e1emem

determme twelve unknowns' Due to redundancy 1n the num'
ber of obtained information, robust extrapolation can be peris the folloWing.

formed. Using the foregoing algorithms, the force vectors are

m(Z)
m(3)
m(4)

Hmx(2, l) Hmz(2, l) Hmx(2, Z) Hmz(2, Z) Hmx(2, 3) Hmz(2, 3) Hmx(2, 4) Hmz(2, 4)
Hmx(3, l) Hmz(3, l) Hmx(3, Z) Hmz(3, Z) Hmx(3, 3) Hmz(3, 3) Hmx(3, 4) Hmz(3, 4)
Hmx(4, l) Hmz(4, l) Hmx(4, Z) Hmz(4, Z) Hmx(4, 3) Hmz(4, 3) Hmx(4, 4) Hmz(4, 4)

fz(l)
fx(2)
fz(2)

n(2)
n(3)
n(4)

Hnx(2, l) Hnz(2, l) Hnx(2, Z) Hnz(2, Z) Hnx(2, 3) Hnz(2, 3) Hnx(2, 4) HnZ(Z, 4)
Hnx(3, l) Hnz(3, l) Hnx(3, Z) Hnz(3, Z) Hnx(3, 3) Hnz(3, 3) Hnx(3, 4) Hnz(3, 4)
Hnx(4, l) Hnz(4, l) Hnx(4, Z) Hnz(4, Z) Hnx(4, 3) Hnz(4, 3) Hnx(4, 4) Hnz(4, 4)

fz(3)
fx(4)
fz(4)

Where Hmx(x1, x2) represents a displacement amount in
x-direction of m marker in a certain depth at a coordinate
xql With a unit force in the x-direction applied to a surface

at a coordinate xq2. Similarly; HnZ(x1, x2) represents a

extrapolated from the CCD image. Even With other measure
ment methods of the present invention using other types of
35 marker con?gurations, for example, the measurement methods are substantially the same.

displacement amount in Z-direction of n marker in a certain
depth at a coordinate xql With a unit force in the Z-direction
applied to a surface at a coordinate xq2.

[D] Transfer Function Used for Reconstructing Force
Vector Distribution

This is a simple multiplication of matrices Where reference
x is 1x8 matrix reference H is 8x8 square matrix, and reference f comprises 1x8 components. Thus, f can be obtained
from observed x by multiplying an inverse matrix of H.

40

Next, a method of obtaining the transfer function Will be
described. In an elastic body having a characteristic shape (asemi-in?nite elastic body, for example), as a function de?ning

Namely, f:inv(H)><X (Equation 1) Where inv represents
inverse matrix (generaliZed matrix inverse).

a force applied to a surface and an internal strain, a function

The matrix form Written With an element is the folloWing.

Where the foregoing equation held in the microscopic regionfx(l)Imx(l, l) Imx(2, l) Imx(3, l) Imx(4, l)fz(l)Imz(l, Z) Imz(2, Z) Imz(3, Z) Imz(4, Z)
fx(2)Imx(l, 3) Imx(2, 3) Imx(3, 3) Imx(4, 3)fz(2)Imz(l, 4) Imz(2, 4) Imz(3, 4) Imz(4, 4)
fx(3) : Imx(l, l) Imx(2, l) Imx(3, l) Imx(4, l)fz(3)Imz(l, Z) Imz(2, Z) Imz(3, Z) Imz(4, Z)fx(4)Imx(l, 3) Imx(2, 3) Imx(3, 3) Imx(4, 3)fz(4)Imz(l, 4) Imz(2, 4) Imz(3, 4) Imz(4, 4)Inx(l, l)Inz(l, Z)
Inx(l, 3)Inz(l, 4)Inx(l, l)Inz(l, Z)
Inx(l, 3)Inz(l, 4)

Inx(2, l)Inz(2, Z)
Inx(2, 3)
Inz(2, 4)
Inx(2, l)Inz(2, Z)
Inx(2, 3)
Inz(2, 4)

Inx(3, l)Inz(3, Z)
Inx(3, 3)Inz(3, 4)
Inx(3, l)Inz(3, Z)
Inx(3, 3)Inz(3, 4)

Inx(4, l)Inz(4, Z)
Inx(4, 3)
Inz(4, 4)
Inx(4, l)Inz(4, Z)
Inx(4, 3)
Inz(4, 4)

m(l)
m(Z)
m(3)
m(4)
n(l)
n(2)
n(3)
n(4)

Where Imx(1,1) and the like represent each element of inv(H) 60 can hold in any regions of the internal portion of the elastic

and represent contribution of m(l) for calculating fx(1).

body has been found as a numerical equation. In this case, a

The important thing is that the number of observed data

matrix H can be obtained by substituting coordinates of ?nely
divided elastic body surfaces and coordinates of internalmust be equal to or more than the number of unknowns When

determining unknoWns by using an inverse matrix de?ned bya transfer function. To solve this problem, the present invention employs tWo layers of differentially colored marker
groups so as to increase the number of independent observed

65

markers into the function.
Here, the numerical equation is a function G by Which theinternal strain can be obtained from the surface stress in theUS 7,659,502 B2
11

12

form of m(x2, y2):G(f(x1), x2, y2), Where f(x1) represents

necessary that forces applied to each sampling point have thesurface stress and m(x2, y2) represent internal strain. For

same magnitude as long as the magnitude of each force is
knoWn. Movement vector of markers may be measured basedon forces having different magnitudes, and later on, the mag

example, When a force is applied to a point 1 in FIG. 7 and

FIG. 8, displacement of marker 2 can be obtained by m(2,y2):G(f(1), 2, y2), Where y2 is a known marker depth.

nitude of movement vector can be normaliZed.

Depending on the shape of elastic body, a matrix H is

obtained using the foregoing equation assuming that an elas

As foregoing, the matrix H can be obtained by simulation

tic body is a semi-in?nite elastic body. It is found that surface
stress cannot be correctly obtained When the equation forsemi-in?nite elastic body is applied for an arbitrary curved

or measurement Where the matrix H is the transfer functionthat associates force information F With information M as tothe behavior of marker (movement information, forexample). The optical tactile sensor comprises a memory

surface such as a hemispherical surface. It is therefore nec
essary to associate a surface stress With an internal strain by

device and a processor. The matrix H obtained is stored in the

any other means.

memory device. A marker image is obtained by a photographing device When an object contacts the transparent elastic

A ?rst method is to associate a surface stress With aninternal strain by numerical simulation. By dividing the sur
face of the sensor into meshes, it is possible to calculate the 20
movement amount of markers When a unit force is applied toeach mesh in x-direction, y-direction and Z-direction by simu
lation.
A second method is to actually apply a force to the surface.

body and an arbitrary force is applied to a surface of a sensor.
A movement vector of marker is measured from the obtained

marker image by the processor. The measured movement
vector of marker is input to the matrix H and calculated by theprocessor, thereby outputting force vector that is applied tothe surface of the elastic body.

25[E] Computation Time Reduction Method

Forces F1, F2, F3, F4 . . . , Fn having knoWn magnitude areapplied to an arbitrary curved surface of elastic body. Movement vectors (Movements of markers caused by each knoWn
force) M1, M2, M3, M4, . . . , Mn of markers as to each force 30

applied are measured and stored. F1 represents three vectors

Flx, Fly, F12 and movement vectors of respective markers

ment information becomes long. This is due to use of moveare given as Mlx, Mly, M12 When these forces are applied. A

matrix H is obtained from the forces having knoWn magni
tude and obtained information (movement vector). The sec
ond method Will be explained in detail.

Here, if the number of elements of a matrix H becomeslarge, the time for calculating force distribution from move35

ment information for all markers When obtaining force
applied to a particular point. In actual fact, in the case of

adopting the previously described algorithm, the H matrix
becomes gigantic, and time is taken in matrix operation forFirstly, numerous sampling points are discretely arranged

equation 1. Giving one example, in the case of a mesh of
100x100, there are 10,000 observation points Which meansthat H matrix becomes a gigantic matrix of l0,000><l0,000.
Generally, in the case of a sensor surface partitioned intoon the surface of elastic body. In one preferable aspect, thesampling points are arranged so as to cover an overall area ofthe surface. In one aspect, numerous discrete sampling points

are arranged (concentrically arranged inplan vieW) according

N><N, since the number of observation points are N squared,
to curvilinear coordinates. In another aspect, the sampling
the siZe of the H matrix becomes N squared by N squared.
points are arranged to provide a grid arrangement in a plan 45
Thus, time of four times N is taken for matrix operation ofview.
equation 1 . It means that this problem is brought to the fore forthe environment type sensor of the present invention that
often comprises a large area surface. Accordingly, it becomes
having knoWn magnitude applied in x-direction, y-direction,
50
and Z-direction With corresponding movement vectors ofnecessary to have a method for shortening the computation
markers When the forces are applied is obtained. In one preftime.

At each sampling point, information that associates forces

erable method, forces having the predetermined magnitude
are independently applied to each sampling point in x-direction, y-direction and Z-direction, and each movement vector 55
of markers is measured and stored. Orientations of x-direction, y-direction and x-direction of force vectors applied on

The proposed method extracts a part of the H matrix andutiliZes the same. As described above, a correspondence relationship for force applied to all lattice points and movementof all markers is described in the H matrix. HoWever, as an

the sampling points are not limited as long as an arbitrary

actual problem, for example, it is possible to ignore the effectforce applied to the surface can be represented by using thesemarker provided that a distance betWeen the force application
points and the marker is suf?cient. If this is done, for example,
in FIG. 7 and FIG. 8, by assuming that it is acceptable to use
Forces applied to each sampling point have knoWn magni
only ?rst to second markers in calculating f(1), and to use only
tude, and in one preferable aspect, a force With constantmagnitude, 100 [gf] for example, is applied to the sampling 65 ?rst to third markers in calculating f(2), it is possible to makethe siZe of the matrix small. A neW matrix in this example ispoint in x-direction, y-direction, and Z-direction, respectively
force vectors.

and movement vectors of each instance are measured. It is not

60

as folloWs.

US 7,659,502 B2
14

13
Original equation 1 f:inv(H) x is as follows:fx(1)Imx(1, 1) Imx(2, 1) Imx(3, 1) Imx(4, 1)fz(1)Imz(1, Z) Imz(2, Z) Imz(3, Z) Imz(4, Z)
fx(2)Imx(1, 3) Imx(2, 3) Imx(3, 3) Imx(4, 3)fz(2)Imz(1, 4) Imz(2, 4) Imz(3, 4) Imz(4, 4)
fx(3) : Imx(1, 1) Imx(2, 1) Imx(3, 1) Imx(4, 1)fz(3)Imz(1, Z) Imz(2, Z) Imz(3, Z) Imz(4, Z)
fx(4)Imx(1, 3) Imx(2, 3) Imx(3, 3) Imx(4, 3)fz(4)Imz(1, 4) Imz(2, 4) Imz(3, 4) Imz(4, 4)

Inx(1, 1)
Inz(1, Z)
Inx(1, 3)
Inz(1, 4)
Inx(1, 1)
Inz(1, Z)
Inx(1, 3)
Inz(1, 4)

Inx(2, 1)
Inz(2, Z)
Inx(2, 3)
Inz(2, 4)
Inx(2, 1)
Inz(2, Z)
Inx(2, 3)
Inz(2, 4)

Inx(3, 1)
Inz(3, Z)
Inx(3, 3)Inz(3, 4)
Inx(3, 1)
Inz(3, Z)
Inx(3, 3)Inz(3, 4)

Inx(4, 1)
Inz(4, Z)
Inx(4, 3)
Inz(4, 4)
Inx(4, 1)
Inz(4, Z)
Inx(4, 3)
Inz(4, 4)

m(l)
m(Z)
m(3)
m(4)
n(1)
n(2)
n(3)
n(4)

15

This equation becomes an equation as follows by ignoring

elements, and the value of matrix elements having a value

contribution at places separated by distance.

fx(1)
fz(1)
fx(2)

0
O
Imx(1, 1) Imx(2, 1)
OO
Inx(1, 1) Inx(2, 1)O
Imz(1, Z) Imz(2, Z) Imz(3, Z)
O
Inz(1, Z) Inz(2, Z) Inz(3, Z)
O
Imx(2, 3) Imx(3, 3) Imx(4, 3)O
Inx(2, 3) Inx(3, 3) Inx(4, 3)OO
Imz(3, 4) Imz(4, 4)OO
Inz(3, 4) Inz(4, 4)fz(2)fx(3) : Imx(1, 1) Imx(2, 1)

fz(3)
fx(4)
fz(4)

smaller than the threshold value are made Zero.

OO

Inx(1, 1) Inx(2, 1)

OOO
Imz(1, Z) Imz(2, Z) Imz(3, Z)
O
Inz(1, Z) Inz(2, Z) Inz(3, Z)
O
Imx(2, 3) Imx(3, 3) Imx(4, 3)O
Inx(2, 3) Inx(3, 3) Inx(4, 3)OO
Imz(3, 4) Imz(4, 4)O
0
Inz(3, 4) Inz(4, 4)

m(l)

m(Z)
m(3)
m(4)
n(1)

n(2)
n(3)
n(4)

30

Further, improvement for the computation shortening

Places With Zero are points separated by distance that shouldbe ignored. Calculation at higher speed can be effectuatedbecause there is no need to compute them. This speed increaseprovides an accelerated effect as the lattice siZe N becomeslarger, as described previously.

method Will be described. With respect to the point of extract
ing a particular region of the image, it is the same as for the35

foregoing computation shortening method. The foregoing
method deals With only information inside the extracted

region While the improved method provides sampling points

This is equivalent to using an extracted surface area con

outside the extracted region for force to be computed. Withtaining a place Where it is desired to obtain force vectors (FIG.
respect to movement of the marker, only information inside10). With the situation in the draWing, in a tWo dimensional
image of the entire contact surface, a region in the vicinity of 40 the extracted region is handled. The sampling points arepoints for taking into consideration the effect of force froma place Where it is desired to obtain force vector distribution
the outside region. That is, movement of a marker inside theis set. Here, in the event that a region in the vicinity of a placeextracted region can be affected to a certain extent by force
Where it is desired to obtain force vectors is set, the neighborapplied outside the extracted region. At the time of reconing region is not necessarily determined based on only adistance in the tWo dimensional image. Namely, a distance 45 structing force vectors based on movement information ofmarkers inside the extracted region, by computing not only
betWeen force application points and a marker is a spatial
force acting inside the extracted region force but also force
distance, and in cases such as Where marker groups are lay

acting outside the extracted region, force acting inside theered inside an elastic body, there may be cases Where it isextracted region is reconstructed While taken into consider
ation force acting outside the extracted region.
Also, force sampling points outside the region are set
surface, contribution of places separated by a distance is notsparsely With distance from the extracted region. This is
alWays small. Therefore another method of shortening com
because it is considered that representation is possible With
putation time is proposed. First of all, a transfer function (aninverse matrix) is obtained by actual measurement or simu 55 feWer sampling points because the effect With becomesslighter With increased distance of separation from the region.
lation. A method for producing this type of transfer function
If description is given based on FIG. 11, a small region cor
has already been described. For example, When looking at
responding to sampling points of 5x5 points is set in an image
elements of a matrix for a transfer function based on actualof a tactile sense section. Sampling points are set densely
measurement, if a particular element of the matrix approaches
Zero, a marker corresponding to the element can be consid 60 inside the small region. Sampling points for force are also

desirable to consider depth With the marker is provided.

50

In the case of a contact surface that is an arbitrary curved

ered to be a marker that can be ignored for the purpose ofarranged outside the small region. Sampling points for force

outside the small region are arranged more sparsely Withobtaining a particular force vector. It is possible to computethe force vector With sections Where the particular element of
distance from the small region. With the example in the draW
the matrix of the transfer function approaches Zero automati
ing, at sites close to the small region, sampling points arecally omitted from the computation. For example, a threshold 65 arranged densely, at the same density as inside the small
representing an index of the extent to Which it is possible toregion, and as separation from the small region increases, theignore in force vector reconstruction is set for the matrix
sampling points are arranged more sparsely.

US 7,659,502 B2
15

16

Then, force vectors for sampling points arranged insidegroup comprised of a plurality of planes extending in theand outside the small region are calculated using movement
information of markers inside the small region. Of the calcu
lated force vectors, only the results for at least some of theforce vectors inside the small region are adopted and saved assame direction, and extending directions and colors thereof
are different betWeen each marker group. The shape of the?nal computation results. With the example in FIG. 11, force
vectors for sampling points for force arranged 3x3 inside theshaped or ?at.
Though the present invention is described based on thecolored markers is not particularly limited, and preferable

examples can be spherical, cylindrical, columnar, strip

small region of 5x5 are saved as ?nal calculation results.

spherical markers as one of preferable aspects, the shape

Sampling points outside the small region and sampling points

and/or arrangement of markers are not limited to the forego

ing. Referring to FIGS. 16 to 19, other shapes and arrange

not adopted inside the small region are discarded. The
extracted small region is then sequentially shifted as obtaining force vectors for the region, so that force vector distribu
tion for the entire measurement region of the sensor is
obtained. In FIG. 11, force vectors for a part of sampling
points of the small region are utiliZed, but it is also possible toutiliZe force vectors for all sampling points inside the small
region. Also, in FIG. 11, force vectors for a plurality ofsampling points for a part of the small region are utiliZed, but
it is also possible to utiliZe only force vectors for one samments of markers Will noW be described. Detail descriptionsof these markers are described in WO02/l 8893 Al and incor

porated herein by reference. Further, the shape and/or
arrangement of markers are not limited to the draWings of thepresent application and WO02/l 8893 Al.
Referring to FIG. 16, colored markers being comprised ofextremely thin cylinders or columns having microscopic
cross sections are shoWn. TWo marker groups are arranged at

different depths from the surface 2. A marker group made up

pling point inside the small region. In FIG. 11, a small region 20 of extremely thin blue cylindrical markers 40 and anothermarker group made up of extremely thin red cylindrical markof 5x5 points is shoWn, but the siZe of the extracted region isers 30 are embedded along the surface 2 and are layered at
not limited. Using this improved method, compared to thedifferent depths from the surface. The markers extend along
above described computation shortening method, the amountof computation may be increased but there is su?icient short
imaginary lines connecting an object coming into contact
ening of the computation time compared to before adopting a 25 With the elastic body and a camera. Arrangement of each

speed increasing method.

marker is not limited to the draWing, and it is possible toprovide three or more groups of marker each having different
colors.
on FIG. 12 to FIG. 15. With FIG. 12, for ease ofdescription of
the improved method the number of marker points is
Referring to an upper vieW of FIG. 17, inclined planeincreased compared to that shoWn in FIG. 7 and FIG. 8. Due 30 markers 300, 400 are arranged in the elastic body 1 in a

Description Will be given for an improved method, based

step-like fashion. In one preferable aspect, parts (a step
shaped interface) of the elastic body 1 constitute markers 300,

to the increased number of marker points, there is a corre

sponding increase in the number of elements in the equations,400. In another aspect, separate plane markers may be embedand for that reason, in FIG. 13, only x, Z directional movements for a marker n corresponding to force for one particular

ded in the elastic body 1. The interface can be divided into tWopoint is shoWn. This is equivalent to a situation Where only 35 surface groups, all surfaces in a group having the same direc

sections corresponding to elements lmx(1,1), lmx(2,1), Imxtion. The surfaces in each group are made the same color (hereone interface 300 is blue, and the other interface 400 is red).

(3,1), lmx(4,1), Imx(1,2), lmx(2,2), lmx(3,2) and lmx(4,2) of

It is possible to acquire observation values containing verticalmarker points is increased.
and horizontal components of force vectors at a particular
The speed increasing method before improvement uses 40 point as information by observation of intensity of the tWo
only movement information of markers that exists close tocolors at that point. By sensing the observed intensity, it ispossible to reconstruct surface distribution of force vectors.
points to Which force being obtained is applied, and as shoWn
matrix inv(H) in equation 3 are extracted and the number of

The surface markers having tWo colors are illustrated in thein FIG. 14, only movement information for markers m1O toupper vieW of FIG. 17, but surface markers having three
ml4 is used. In this respect, if the speed increasing method
after improvement is illustrated, it is as shoWn in FIG. 15. 45 colors may be used. As shoWn in the loWer vieW of FIG. 17,

using so called pyramid manufacturing Where microscopic

That is, not only F8, but also F2, F5, P9, P11 and F14 arearranged as force sampling points. In the computed force

cubes are gathered at a bottom surface, if three groups of
surfaces facing in the same direction are respectively madevectors, only force vectors for PX8 and F28 are utiliZed.

[F] Other Embodiments of Marker Group50

the same color (for example, red, green and blue), it is pos
sible to respectively obtain degrees of freedom for force act
ing in a horizontal direction on a contact surface as intensity

As for a tactile portion of optical tactile sensor, in preferredratios for three colors, and force acting in a vertical direction
using a total intensity of the three colors.
embodiments, a plurality of groups of markers are embeddedin the transparent elastic body, each group of markers being
Referring to FIG. 18, tWo marker groups (a marker groupmade up of a large number of markers, markers constituting 55 comprising a plurality of thin red strips arranged in a roW anddifferent marker groups having different colors for eacha marker group comprising a plurality of thin blue strips
group, and the marker groups having a different spatial
arranged in a roW) are aligned so that respective markers arearrangement. As an example of this differing spatial arrange

orthogonal to each other, but the spatial arrangement relation

ship betWeen the plurality of marker groups is not limited. It
ment, a plurality of marker groups are arranged in a layered
manner inside the elastic body. As an example of layered 60 is also possible for the tWo sides of the strips constituting themarker to have different colors. In the draWing, side portions
markers, the markers constituting the marker groups areof the strip markers extend along an observation direction but
microscopic spherical particles and the spherical markers
the side portions of the strip markers may be inclined to an
constituting the marker group for each layer have different
observation direction.
colors from each other. As another example of this differing
FIG. 19 shoWs a sensing part having a plurality of planespatial arrangement, a plurality of marker groups are arranged 65
so as to intersect each other. As still another example of thismarkers. The plane markers are normally concealed by con
cealment markers and each plane marker is partitioned into a
differing spatial arrangement, each marker group is a plane<19)РОССИЙСКАЯ ФЕДЕРАЦИЯ

U <11)(13R
� 37"1 6�6 ) С2(51) мпкGOJL 5/16 (2006.01)

ФЕДЕРАЛЬНАЯ СЛУЖБАПО ИНТЕЛЛЕКТУАЛЬНОЙ СОБСТВЕННОСТИ,ПАТЕНТАМ И ТОВАРНЫМ ЗНАКАМ
(12) ОПИСАНИЕ ИЗОБРЕТЕНИЯ К ПАТЕНТУ

(21), (22) Заявка: 2006108382/28, 03.03.2.005(24) Дата начала отсчета срока действия патента:03.03.2005(30) Конвенционный приоритет:16.06.2004 JP 2004-177786(72) Автор(ы):ТАТИ Сусуму (JP),КАБАКАМИ Наоки (JP),КАДЗИМОТО Хироюки (JP),КАМИЯМА Казуто (JP),МИЗОТА Теруказу (JP):;ос:(73) Патентообладатель(и):ТОУДАИ ТЛО, ЛТД. (JP)(43) Дата нубликации заявки: 27.09.2007(45) Опубпиковано: 27.10.2009 Бюл. № 30(56) Список документов, цитированных в отчете опоиске: WO 0218893 Al, 07.03.2002. JP 7067020 А,
10.03.1995. JP2001141660 А, 25.05.2001. SU
1288730 Al, 07.02.1987. RU 10905 Ul,
16.08.1999.

......
cn00(85) Дата перевода заяв,ш РСТ на национальную
фазу: 16.03.2006
N

u
<О
СХ)
<О

....

cn

(86) Заявка РСТ:JP 2005/003650 (03.03.2005)(87) Публикация РСТ:wo 2005/124305 (29.12.2005)
Адрес для переписки:129090, Москва, ул. В.Спасская, 25, стр.З,ООО "Юридическая фирма Городисский иПартнеры", пат.пов. Ю.Д.Кузнецову,рег.№ 595(54) ОПТИЧЕСКИЙ ТАКТИЛЬНЫЙ ДАТЧИК
(57) Реферат:Настоящее изобретение относится коптическому тактильному датчику, а именно ксодержащую прозрачный гибкий корпус имаркеров,множество

расположенных

вкорпусе, и множество устройств формирования
изображения

длябольшей 1птощади. Техническим результатом

маркера путем фотографирования поведения
маркеров, когда объект контактирует сзаявленного изобретения является получение

тактильной

поверхностью

тактильному датчику,используемому дляполучения сил, приложенных к относительно

информации
изображения

исформирование

целого

помощью
маркеров,позволяющего измерять распределение силына большой rшощади. Оптический тактильный
часгь,
содержит
датчик
тактильную

гибкого

корпуса.

получения

Способ

изображения

прозрачного

использования

оптического тактильного датчика содержит
этапы получения частичного изображения

путем фотографирования частичных областей
А, В, С и D прозрачного гибкого корпуса сСтраница: 1

(19)

в

KOREAN INTELLECTUAL PROPERТY OFFICE
KOREAN PATENT ABSTRACTS

1020070036015 А(11) PuЬlication number:(43) Date of puЬlication of
application:(21) Application
number:(22) Date of filing:(ЗО) Priority:(51) lnt. CI:

1020067005120
13.03.2006
2004 2004177786 JP 16.06.2004(71) Applicant:(72) lnventor:

GOlL 1/24 (2006.01);
GOlL 5/ 16 (2006.01);
G06F 3/03 (2006.01);
G06F 3/00 (2006.01);

02.04.2007

TOUDAI TLO, LTD.
TACHI SUSUMUКАWАКАМI NAOКIКAJIMOTO HIROYUКI
КАМIУАМА KAZUTOMIZOTA TERUKAZU(54) OPТICAL TACТILE SENSOR(57) Abstract:
An optical tactile sensor and an information integrating
method сараЫе of measuring large area force vector
distribution. The integrating method of а marker image in an
optical tactile sensor comprising а tactile section consistingof а transparent resilient body (1) and а plurality of markers(З, 4) provided therein, and а plurality of imaging devices(6, 6) for acquiring а marker image Ьу photographing the
behavior of the marker when an object (5) touches thetactile face (2) of the transparent resilient body (1), themethod comprising а step for acquiring а partial image
Ьу photographing the partial regions А, В, С and D of
the transparent resilient body (1) using the plurality ofimaging devices (6, 6) such that each imaging device (6)has an overlapped photograph region (11), and а step for
integrating the partial images acquired Ьу each imaging
device (6) such that the identical markers in the overlapped
photograph region match.

i
А

1

в

1

;
:
f
--------------------------- ---- - - - -------1

11�
1

1

1

:- ! �
:у

1

с

1

1

11

о

1

------------�-------------1

1© KIPO & WIPO 2007

This Facsimile First Page has Ьееп artificially created from the Korean Patent Abstracts CD Rom