US 20030057973A1(19) United States(2) Patent Application Publication (10) Pub. No.: US 2003/0057973 A1
Nojima et al.

(43) Pub. Date:(54) SMART-TOOLAND METHOD OF

Mar. 27, 2003

(52) U.S. Cl. … 324/753

GENERATING HAPTIC SENSATIONTHEREOF(57)(76) Inventors: Takuya Nojima, Musashino-shi (JP);Dairoku Sekiguchi, Meguro-ku (JP);Masahiko Inami, Katsushika-ku (JP);Kunihiko Mabuchi, Itabashi-ku (JP);Susumu Tachi, Tsukuba-shi (JP)ABSTRACTThe present invention provides a smart tool for real-timemeasurement of a changing environment and real-time dis
play of such information to the user, thereby supporting theuser for example in surgery. Visual information in thevicinity of the tool tip is displayed to the user of the tool as

Correspondence Address:
OLIFF & BERRIDGE, PLC
P.O. BOX 19928

haptic information. The smart tool comprises a tool (surgicalknife), a light-receiving element for receiving the light

ALEXANDRIA, VA 22320 (US)(21) Appl. No.:
10/213,087(22) Filed:Aug. 7, 2002(30)
Foreign Application Priority Data
Aug. 8, 2001 (JP)...................................... 2001–240127
Publication Classification(51) Int. Cl." ........................ G01R 23/17; G01R 31/308

reflected by an object in the vicinity of the tool tip, a decision
portion for deciding the brightness and/or color of the object
in the vicinity of the tool tip based on the output from said
light-receiving element, a repulsive force calculation portion
for calculating the repulsive force to be applied to said tool
based on the output from said decision portion, a motor
driving circuit for applying repulsive force to said tool basedon the output from the repulsive force calculation portion, a
motor, and a wire.

16

17

CIRCUIT FOR

DRIVING MOTOR
20

REPULSIVE FORCECALCULATIONCIRCUIT FOR DRIVINGPORTION

LIGHT-LUMINOUS
ELEMENTLIGHT-RECEIVINGPatent Application Publication Mar. 27, 2003 Sheet 1 of 8

US 2003/0057973 A1

F|G. 1
HAPTIC(SENSE OF FORCE)DISPLAYENVIRONMENTAL
INFORMATION/HAPTICSENSATION(SENSE OF FORCE)

CONVERSION PORTION

PORTIONPatent Application Publication Mar. 27, 2003 Sheet 2 of 8

US 2003/0057973 A1

F| G. 2Si
OPERATE TOOL

EXAMPLE: ACTUATOR (KNIFE) TRANSFIXES OR DISSECTS OBJECTMEASURE ENVIRONMENTAL INFORMATION USING SENSOR
FOR MEASURING TOOL TIP ENVIRONMENTEXAMPLE: LIGHT-SCATTERING INTENSITY, COLOR, DISTANCE TO INTERFACE,ELECTRIC RESISTANCE, ELECTRIC FIELD INTENSITY, MAGNETIC
FIELD INTENSITY, TEMPERATURE, PRESSURE, VIBRATION, SOUND,HARDNESSCALCULATE HAPTIC SENSATION (REPULSIVE FORCE) TO BE
GENERATED BASED ON ENVIRONMENTAL INFORMATION

EXAMPLE: - GENERATE REPULSIVE FORCE WHEN LIGHT-SCATTERINGINTENSITY CHANGES- GENERATE REPULSIVE FORCE WHEN COLOR. CHANGES— GENERATE REPULSIVE FORCE WHEN DISTANCE TO
INTERFACE BECOMES SHORT. SIZE OF FORCE IS
INVERSELY PROPORTIONAL TO DISTANCE,DRIVE HAPTIC (SENSE OF FORCE) DISPLAY TO APPLYCALCULATED HAPTIC SENSATION (REPULSIVE FORCE) TO TOOL
EXAMPLE: PULL ACTUATOR (KNIFE) WITH MOTORS5APPLY HAPTIC SENSATION (SENSE OF FORCE) TO TOOL

Patent Application Publication Mar. 27, 2003 Sheet 3 of 8

US 2003/0057973 A1

F| G. 3ASJZE OFREPULSIVEFORCELIGHT-SCATTERINGINTENSITY (COLOR)(WHITE)(YELLOW)INTERFACE

F| G. 3B
SIZE OFREPULSIVEFORCE

H--INTERFACEDISTANCE

Patent Application Publication Mar. 27, 2003 Sheet 4 of 8

US 2003/0057973 A1

F|G. 416
CIRCUIT FOR

DRIVING MOTORCIRCUIT FOR DRIVING
LIGHT-LUMINOUS

REPULSIVE FORCECALCULATION,PORTION

BRIGHTNESS(COLOR)JUDGEMENTPORTION22

Patent Application Publication Mar. 27, 2003 Sheet 5 of 8

US 2003/0057973 A1

Patent Application Publication Mar. 27, 2003 Sheet 6 of 8

US 2003/0057973 A1

F|G. 637
CIRCUIT FOR

36

DRIVING MOTOR

REPULSIVE FORCECALCULATION PORTION

38

ELECTRIC RESISTANCEJUDGEMENT PORTION

30

N

Z iºº ºWA

35

34

Patent Application Publication Mar. 27, 2003 Sheet 7 of 8

US 2003/0057973 A1

F| G. 737
CIRCUIT FOR
DRIVING MOTOR REPULSIVE FORCE

36
35

CALCULATION PORTION
38
ELECTRIC RESISTANCEJUDGEMENT PORTION

30
31

ºsN

N

32

N NZ NWA

34

Patent Application Publication Mar. 27, 2003 Sheet 8 of 8

US 2003/0057973 A1

F|G. 843

DISTANCE

CHANGE/HAPTIC SENSATIONCONVERSION PORTION44

DISTANCE CHANGE
DETECTION PORTION

40

RANGE FINDERLASER POINTER

41

US 2003/0057973 A1

SMART-TOOL AND METHOD OF GENERATINGHAPTIC SENSATION THEREOF[0001] The present application claims priority of Japanese
patent application No. 2001-240127 filed Aug. 8, 2001, thedisclosure of which is fully incorporated by reference herein.
BACKGROUND OF THE INVENTION[0002] 1. Field of the Invention[0003] The present invention relates to a smart tool
wherein a sensor, a tool and an information display device
are integrated into one device, which realizes haptization of
information by displaying non-haptic information as haptic
information, and the method of generating haptic sensationof the smart tool.

[0004] 2. Description of the Related Art[0005] Technologies related to haptic (meaning “sense offorce”; hereinafter the same) display are being frequently
used in the general society, and many researches are being
conducted relating to the application of such technologies.
Such researches are presently still directed at realizing thedisplay of haptic sensation that is close to real sensation,
namely, which can be hardly distinguished from the sense of
touching an object in reality. This type of display may benecessary for simulations in the medical field or for gamesin the amusement field. However, due to the complexity ofhaptic sensations, there is still no all-purpose haptic displaythat closely presents haptic sensations of the real world.
Therefore, application of haptic display technologies in thegeneral society should not be directed at perfectly displaying
haptic sensations but at integrating with reality.
[0006] The following points can be said from the view
point of the origin of haptic displays and virtual reality:[0007] 1. Haptic sensation is “impedance information” relating to an object when touching the object,
and the method of displaying such information is a“haptic display;”[0008] 2. “Virtual reality” is the “essence of reality,”and “information” essential to a certain purpose must
be displayed to the user; and[0009] 3. “Augmented reality” is different from “reality,” but is the same in that it displays necessary“information” to the user. However, such informa

tion is commonly too much inclined toward visual
information.

[0010] Conventional haptic displays present haptic infor
mation by setting appropriate impedance to a virtual object.
“Information” conveyed by the impedance is information
related to the “impedance of the object being touched.” Theflow of conventional “haptic reality” strives to completely
reproduce this information. However, in the present invention, necessary information other than the impedance of theobject is conveyed by the impedance information. Thereby,
augmented reality using haptic sensation can be realized.
“Touchable information” is realized, i.e., non-haptic infor
mation is presented through haptic sensation.
[0011] This, namely the augmented haptics on informa
tion, will be described below. According to conventional
methods, a sensor attached to (or in the surrounding of) atool is used to obtain information on the environment of theMar. 27, 2003

tool; the obtained information is displayed visually or acoustically to the user, who then performs processing of theinformation in his mind. However, in the present invention,

information is displayed haptically (impedance information).
[0012] A currently found example, the fire alarm cover,shows this clearly. This type of cover basically functions toprevent operational mistakes, but also displays, with maxi
mum impedance, information meaning “Do not press thisbutton except in situation of fire”. This method can reducethe burden of information processing on the user, and thenumber of mistakes caused thereby. Information meaning“Do not press” is displayed through impedance.
[0013] In the example described above, impedance isdisplayed using an actual object, but in the medical field, itshould be possible to display information relating to dan
gerous areas that should not be touched during surgicaloperation through impedance. Dangerous tools used forsurgery such as knives and needles are moved by thedoctor’s hand so as not to injure organs in the patient’s bodythat are relevant to the patient’s life, but actually, theabove-mentioned dangerous area changes dynamically. This
dynamically changing dangerous area is measured in realtime, and information meaning “Do not touch” is displayedin real time through impedance. Thereby, the course of
information processing normally starting from visual recognition, judgment of dangerous area within the brain, andcontrol of the body so as not to touch the area can becommissioned to an external device.

[0014] In this case, it should be noted that in all cases, theuser should take initiative in acting. It is not an object of thepresent invention to control the operation of the user exactly
in the way the system designer has simulated it, but merely
to provide “information” through “impedance,” and to
reduce the burden of information processing on the user.
This objective is not restricted to the medical field.
[0015] Factors essential to the realization of the present

invention are: a) function as a normal tool; b) mechanism ofreal time measurement; and c) mechanism of real time

display. A tool having all of these three functions is necessary, and this tool will be called herein a “smart tool.” By
using this smart tool, the flow of information that conventionally passed through the mind of the user forms a loop via
the environment, the tool, and the hand of the user, thereby
allowing reduction of the burden on the user. In other words,
the “tool”knows” the information and passes on the infor
mation to the user through a method unique to the tool.
SUMMARY OF THE INVENTION[0016] The present invention was made from the above
described perspectives, and aims at providing a smart tool
wherein a sensor, a tool and an information display device
are integrated into one device, and information is presented
to the user in the form of impedance.
[0017] The smart tool is a new type of Augmented Reality(AR) The smart tool is made of real-time sensing devices
and a haptic display. The sensor senses real environmentsthat change dynamically, and displays that information to theuser through haptic sensation. In other words, the smart tool
makes it possible to “touch” the dynamic information of realenvironment in real time.

US 2003/0057973 A1[0018] An object of the present invention is to provide asmart tool that measures a dynamically changing environ
ment in real time for example in a surgical operation, anddisplays the information in real time to the user so as tosupport the user.
[0019] Another object of the present invention is to pro
vide a smart tool that can “touch” the interface between two

different liquids, for example.
[0020] The present invention is a smart tool for displaying
non-haptic information relating to the environment of a tool
as haptic information to the user of said tool, comprising:said tool; a sensor for detecting the environment surrounding
said tool; an environmental information measurement por
tion for gaining environmental information on said tool
based on the output from said sensor; an environmental
information/haptic sensation conversion portion for convert
ing the gained environmental information to haptic sensation; and a haptic display for conveying the converted haptic
sensation to said tool.

[0021] The present invention is a smart tool for displaying
visual information in the vicinity of the tip of a tool as haptic
information to the user of said tool, comprising: said tool; alight-receiving element for receiving the light reflected by anobject in the vicinity of the tip of said tool; a decision portion
for deciding the brightness and/or color of the object in thevicinity of the tip of said tool based on the output from said
light-receiving element; a repulsive force calculation portion
for calculating the repulsive force to be applied to said tool
based on the output from said decision portion; and a haptic
display for applying repulsive force to said tool based on the
output from said repulsive force calculation portion.
[0022] The present invention is a smart tool for displaying
information on the interface between liquids in the surrounding of a tool as haptic information to the user of said tool,comprising: said tool; a first electrode attached to the tip ofsaid tool; a second electrode placed within a liquid; anammeter for measuring the value of the electric current
flowing through said first electrode and said second elec
trode; a decision portion for deciding the electric resistancebased on the electric current value measured by said amme
ter; a repulsive force calculation portion for calculating therepulsive force based on the gained electric resistance; anda haptic display for applying repulsive force to said tool
based on the output from said repulsive force calculation
portion.
[0023] The present invention is a smart tool for displaying
information on the interface between liquids in the surrounding of a tool as haptic information to the user of said tool,comprising: said tool; a first electrode and a second elec
trode attached to the tip of said tool; an ammeter formeasuring the value of the electric current flowing throughsaid first electrode and said second electrode; a decision

portion for deciding the electric resistance based on theelectric current value measured by said ammeter; a repulsive
force calculation portion for calculating the repulsive force
based on the gained electric resistance; and a haptic display
for applying repulsive force to said tool based on the output
from said repulsive force calculation portion.
[0024] The present invention is a smart tool for displaying
information on the surface of a distant object as haptic
information to the user of a tool, comprising: a range finder

Mar. 27, 2003

for measuring the distance to said object; a distance change
detection portion for detecting changes in said distance; aconversion portion for converting changes in the detecteddistance to haptic sensation; and a haptic display for apply
ing haptic sensation based on the output from said conversion portion.
[0025] The present invention is a smart tool for displaying
information on the surface of a distant object as haptic
information to the user of a tool, comprising: a lightreceiving portion for receiving the light reflected by thesurface of said object; a detection portion for detecting thestate of said surface based on the intensity of the lightreceived by said light-receiving portion; a conversion por
tion for converting the detected surface state to haptic
sensation; and a haptic display for applying haptic sensation
based on the output from said conversion portion.
[0026] The present invention is a smart tool for displaying
information on a distant object as haptic information to theuser of a tool, comprising: a proximity sensor for detecting
information relating to an object existing in the vicinity; aconversion portion for converting information on thedetected object to haptic sensation; and a haptic display for
applying haptic sensation based on the output from said
conversion portion.
[0027] The present invention is a haptic sensation genera
tion method of a smart tool, comprising the steps of: anoperator operating said tool; measuring the environment in
the surrounding of said tool and gaining environmental
information other than haptic sensation; calculating thehaptic sensation to be applied to said tool based on said
environmental information; and applying the calculated hap
tic sensation to said tool.
BRIEF DESCRIPTION OF THE DRAWINGS[0028] FIG. 1 shows the structure of the smart toolaccording to Embodiment 1;[0029] FIG. 2 is a flow chart of the operation of Embodi
ment 1;[0030] FIG. 3 is a view explaining the operation of
Embodiment 1;[0031] FIG. 4 shows the structure of the smart toolaccording to Embodiment 2;[0032] FIG. 5 is a view explaining the operation of
Embodiment 2;[0033] FIG. 6 shows the structure of the smart toolaccording to Embodiment 3;[0034] FIG. 7 shows the structure of the smart toolaccording to a variation of Embodiment 3; and[0035] FIG. 8 shows the structure of the smart toolaccording to Embodiment 4.
DESCRIPTION OF THE PREFERREDEMBODIMENTS[0036] Embodiment 1.
[0037] FIG. 1 shows the concept of the smart tool accord
ing to an embodiment of the present invention.

US 2003/0057973 A1

Mar. 27, 2003

[0038] In FIG. 1, numeral 1 denotes a tool operated by theuser, 2 denotes a sensor for detecting the environment of the[0052] Repulsive force is generated when the dis

information measurement portion for gaining informationon the environment of tool 1 based on the output from sensor
2, 4 denotes an environmental information—haptic sensa[0053] As shown in FIG. 3 (a), when the output fromsensor 1 (or a parameter calculated based on this output)

tance to the interface becomes short. The size of theforce is inversely proportional to the distance.

tool (especially the tool tip), 3 denotes an environmentaltion (sense of force) conversion portion for converting thegained environmental information to haptic sensation (senseof force), and 5 denotes a haptic (sense of force) display forconveying haptic sensation (sense of force) to tool 1.
[0039] Sensor 2 is attached in the vicinity of the portion oftool 1 that is to operate on the object. For example, if tool
1 is in the form of a rod with the tip touching the object,sensor 2 is provided on the rod tip. Information gained by
sensor 2 is non-haptic information. In other words, the smart
tool according to the present invention gives feedback to thetool user regarding non-haptic information in the form of
haptic sensation. Haptic sensation means information thatexceeds a predetermined threshold value, a certain size of
repulsive force may be generated. The portion of the thresh
old value corresponds for example to the interface. It is also
possible to cause the generated repulsive force to increase
when the output from sensor 1 nears the predeterminedthreshold value. For example, a repulsive force that isinversely proportional to the distance to the interface isgenerated. As shown in FIG.3(b) by dotted lines, it is also

possible to set a maximum limit to the size of the repulsive
force.

[0054] S4: Haptic (sense of force) display 5 is driven so as
to apply the calculated haptic sensation (repulsive force) tothe user senses with his sense of touch. The sense of touch

tool 1.

presented by the smart tool includes a repulsive force
working against the force of the user operating the tool, avibration of the tool itself, a heat generation of the tool itself,[0055] For example, by pulling the actuator (knife) with aan electric current (electric shock), and a sense of the tool
itself bending (a repulsive force and a force crossing thedirection of the force applied by the user).
[0040] Conventionally, users also used the feeling fromthe tool, namely the sense of force, similarly to the visualand auditory senses as a source of information. The smart
tool aims at providing a sensor on a tool that is used for a
specific operation and at thereby actively displaying infor
mation gained from the sensor to the user in the form of
information relating to the sense of force.
[0041] Now, the operation of Embodiment 1 will be
explained with reference to FIG. 2.
[0042] S1: The operator operates tool 1. If tool 1 is asurgical knife, the operator transfixes or dissects the object
with tool 1.

[0043] S2: A sensor 2 for measuring the environment of
the tip of tool 1 is used to measure the environmental
information. Examples of sensor 2 and the environmental
information measured therewith are shown below:[0044] optical sensor (light-emitting diode combinedwith photodetector): light-scattering intensity, color,distance to interface[0045] electrical sensor (ammeter, voltmeter): elec

tric resistance, electric field intensity[0046] magnetic field sensor: magnetic field intensity[0047] temperature sensor (thermocouple): temperature[0048] pressure sensor: pressure, vibration, sound,hardness[0049] S3: Based on the environmental information, thehaptic sensation (repulsive force) to be applied to tool 1 iscalculated, for example as below:[0050] Repulsive force is generated when the lightscattering intensity changes.
[0051] Repulsive force is generated when the color
changes.

motor, repulsive force working against the force of the user
operating the tool is applied. By rotating a weight with the
motor, vibration can be applied to the tool itself. By ener
gizing a heater within the tool, the tool itself can generate
heat. It is possible to apply a weak electric current to the tool.
It is also possible to apply a repulsive force and a force
crossing the direction of the force applied by the user to thetool to generate a haptic sensation as if the tool itself werebending.

[0056] S5: Haptic sensation (sense of force) is applied tothe tool.

[0057] The smart tool is characterized in that a sensor, atool and an actuator are integrated in one device, and direct
feedback on the information of the tool is given to the tool
via the sensor and actuator.

[0058] The smart tool has the following advantages:[0059] 1) The existing skills of the operator can beused almost fully in their current state because thesame or similar tools as the operator usually uses canbe used.

[0060] 2) By providing a real-time environment mea
surement sensor on the tool, the tool can be used ina dynamically changing actual environment.

[0061] 3) By providing the tool on a sense of force

display, instinctive display of information using
sense of force is possible.

[0062] 4) Environmental information in the surrounding of the tool is collected and sense of force isdisplayed through the tool itself, so the environmen
tal position of the collected information is consistent
with the position of the tool to be controlled, whichis easy to understand. If the positions were notconsistent, coordinate conversion processing to
make the coordinates of the environment and toolconsistent would be necessary.
[0063] Embodiment 2.
[0064] Embodiment 2 of the present invention is a smart
tool directed at real time surgery support. The tool used inthis smart tool is a knife. When dissecting the human bodyUS 2003/0057973 A1

using this smart tool, important tissue within the body suchas the artery is set in advance as a dangerous area; thepositional relationship between such tissue and the tool tip
is measured in real time; and the measured information ispresented to the operator through sense of force.
[0065] FIG. 4 shows a smart tool according to Embodi
ment 2 of the present invention.
[0066] 11 denotes the tool which is a knife; 12 denotes anoptical fiber that guides the light reflected by the object(human body) in the vicinity of the tip of knife 11 to alight-receiving element 13; 14 denotes a brightness (color)decision portion for deciding the brightness or color of theobject in the vicinity of the tip of knife 11 based on the lightguided by optical fiber 12; 15 denotes a repulsive force
calculation portion for calculating the repulsive force to be
applied to knife 11 based on the output from said brightness(color) decision portion 14; 16 denotes a motor driving

circuit for driving a motor 17 based on the output from
repulsive force calculation portion 15; 17 denotes a motor
for applying repulsive force to knife 11 by pulling a wire 18;
18 denotes the wire for conveying the driving force of motor
17 to knife 11; 19 denotes a case for housing knife 11; 20
denotes a circuit for driving the light-receiving element; 21
denotes a light-emitting element such as a light-emitting
diode; and 22 denotes an optical fiber for guiding the lightfrom the light-emitting element to the vicinity of the tip ofknife 11. The smart tool of the present invention is structured
to only generate repulsive force away from a structurally set
dangerous area. The smart tool operates as shown in FIGS.
2 and 3. For example, when the tip of knife 11 is about toharm an important organ, motor 17 generates a braking force
in the form of impedance information to inform the user ofa dangerous area and at the same time stops knife 11 so as
not to harm the important organ.
[0067] Sensors 12, 13, 21 and 22 are required to: 1. workin real time; and 2. take up as little space as possible (as they
are to be incorporated in the tool). In Embodiment 2, thesesensors are not required to be highly precise or have highresolution.

[0068] Now, a detailed operation of the smart tool will be
described with reference to FIG. 5. In this case, a boiled egg
is used instead of the human body as the object, using theegg as an example of the human body, where the egg yolk
is set as a dangerous area denoting important tissue in thehuman body. In FIG. 5, W denotes the egg white and Y
denotes the egg yolk.
[0069] A force F is applied to the smart tool to cause knife
11 to intrude into the inside of the egg (FIG. 5(a)). Whenknife 11 intrudes further, the actuator (made of a motor 17
and a wire 18) applies a braking force immediately before
egg yolk Y (R in FIG. 5 (b)). Accordingly, knife 11 does not
intrude into egg yolk Y but stops immediately before theyolk. To be more accurate, the operator of knife 11 isinformed that the tip of the knife has reached the area where
intrusion is prohibited, and stops the intrusion.
[0070] As in the case in FIG. 5(a), when the operator
moves the tool in an area sufficiently far away from thepredetermined dangerous area Y, the sense of force display
generates no load. However, as in the case in FIG. 5(b),when real time sensor 13 provided on tool 1 senses that thetool has neared the predetermined dangerous area Y, it

Mar. 27, 2003

generates a load via sense of force displays 17 and 18
according to predetermined conditions, and displays to theuser the distance relationship to the dangerous area via senseof force, which is an intuitive means of displaying informa
tion to the user. If repulsive force working away from thedangerous area is used as the generated load, not only
information display is gained but there is also the advanta
geous practical effect of reducing the possibility of harming
important tissue by the tool intruding into the dangerous&l?é à.

[0071] The smart tool displays to the user via impedance
that the tool tip has reached the dangerous area, so importantorgans are not harmed by the knife tip.
[0072] The smart tool can also be applied to cases wherea needle or scissor is used as the tool instead of knife 11.

[0073] Embodiment 2 of the smart tool as described above

calculates linear information such as the distance between

the tool and a predetermined area, and, based thereon,
performs linear control on the tool. It is also possible toperform further control by pre-setting the optimal track,
speed and/or size of force according to which the tool should
move, calculating the difference between the actual track,
speed and/or size of force of the tool and the optimal setting,and displaying haptic sensation (sense of force) to the tool
based on the calculated difference.

[0074] The above variation only generates the track, speed
and size of force that are “optimal” based on a purely
physical model, and displays information based thereon to
the user.

[0075] Embodiment 3.
[0076] The smart tool according to Embodiment 3 is used
as an interface for “touching” the interface between two
liquids. Usually, nobody could feel the interface between
two liquids, because they are liquids. However, using thistool, based on the smart tool technology, the sensor on thetool senses the interface between two liquids and displaysthat information through haptic sensation to the user. The
user could write something on the interface between two
liquids using this tool.
[0077] FIG. 6 shows the structure of the smart tool.
[0078] 30 denotes a rod for intruding into water WA andoil O and informing the user of the interface; 31 denotes anelectrode provided on the tip of rod 30; 32 denotes anelectrode placed within water WA; 33 denotes an ammeterfor measuring the value of the electric current flowing
between electrodes 31 and 32; 34 denotes a decision portion
for deciding the electric resistance based on the electric
current value; 35 denotes a repulsive force calculation
portion for calculating the repulsive force based on theelectric resistance; 36 denotes a motor driving circuit for
driving motor 37 based on the output from repulsive force
calculation portion 35; and 37 denotes a motor for applying
repulsive force to rod 30 via wire 38.
[0079] The smart tool performs the operation as describedfor Embodiments 1 and 2.

[0080] In the device shown in FIG. 6, a user (not illus
trated) holds tool 30 in his hand. When the user moves thetool from the oil layer to the water layer, electric current
flows between electrodes 31 and 32, which is decided byUS 2003/0057973 A1

ammeter 33 and electric resistance decision portion 34, and,based on the decision results, the repulsive force is calcu
lated and motor 37 applies the repulsive force to rod 30.
Accordingly, the user can feel a solid interface between
water WA and oil O.

[0081] Furthermore, it is shown that the user can draw
pictures on the interface. At first, the sensor on the tool
detects the interface between oil and water containing phe
nolphthalein. The sensor has an electrode, which can cause
electrolysis. If the user moves the tool with very weak force,the tool would not go through the interface and the user
could not write anything. However, if the user moves thetool with slightly stronger force, the electrode goes throughto the water layer and causes electrolysis. The electrolysis
changes the acidity so that the phenolphthalein changes its
color to red.

[0082] It is described above that the smart tool can “touch”
the interface. The smart tool can change the hardness of theinterface between the oil layer and the water layer. However,just by changing the control algorithm of the system, it ispossible not only to change the hardness of the environment,
but also the viscosity or any other physical parameter.
[0083] In Embodiment 3 described above, one electrode32 was put into the liquid, and the other electrode 31 was
attached to tool 30. It is also possible to attach bothelectrodes 31, 32 to tool 30, as shown in FIG. 7.

[0084] Through the structure as shown in FIG. 7, theintrusion length of electrodes 31, 32 into the liquid shows arelatively smooth proportional relationship with the resis
tance value between electrodes 31, 32, which is advanta

geous in measuring the intrusion length into the liquid withthe sensor.

[0085] The resistance value between the two electrodes 31
and 32 inside the liquid depends upon the length of theelectrode inside the liquid and the distance between theelectrodes. The resistance value is approximated by roughinverse proportion to the length of the electrodes inside theliquid and rough proportion to the distance, for example.
Therefore, in the case in FIG. 6, the resistance changes not
only according to the intrusion volume of electrodes 31 and
32 into the liquid but also according to the distance between
electrodes 31 and 32, so it is relatively difficult to decide thevolume of the electrodes into the liquid from the sensor
value. However, in the case in FIG. 7, it is possible to holdthe distance between electrodes 31, 32 constant. Therefore,

there is only one factor that has an effect on the sensor
output, thereby enabling relative accurate measurement of
the resistance value depending only on the intrusion volume.
[0086] Embodiment 4.
[0087] The smart tool according to Embodiment 4 of thepresent invention is used as an interface for knowing thestate of the surface of a distant object.
[0088] FIG. 8 shows the structure of the smart tool.
[0089] 40 denotes a laser pointer for radiating a narrowlaser beam onto the surface of a distant object; 41 denotes arange finder for measuring the distance to the portion
radiated by the laser beam; 42 denotes a distance change
detection portion for detecting any distance change; 43
denotes a distance change/haptic sensation conversion por
tion for converting a distance change to haptic sensation;

Mar. 27, 2003

and 44 denotes a vibrator (voice coil) for causing laser

pointer 40 to vibrate. Range finder 41 measures the distancefrom laser pointer 40 to the light spot of the laser beam.
Distance measurement is performed for example by mea
suring the time until the laser beam is reflected and comesback.

[0090] In the smart tool, when the user holds laser pointer
40 and moves it to a certain object, the distance to the objectchanges. When this distance change becomes a predeter
mined value or more, distance change/haptic sensation con
version portion 43 generates a drive signal for vibrator 44.
Then, laser pointer 40 vibrates and conveys a sensation to
the user as if he were touching the surface with a rod.
[0091] Furthermore, it is also possible to measure thesurface state (roughness) using light-scattering intensity
instead of the distance change and generate haptic sensation
based thereon.

[0092] Another alternative is to use a proximity sensor
instead of range finder 41, and apply vibration to the tool
when an obstacle or the like nears the vicinity of the tool. An
example of a proximity sensor is one that radiates radio
waves or ultrasonic waves and receives the reflected waves.

[0093] Furthermore, laser pointer 40 is not an essential
element, and it may be omitted.
[0094] The embodiment of the present invention may be
applied to tools for supporting blind people, for example. By
incorporating the smart tool into the walking stick, the user
can know uneven places on the road such as stairs and steps
through vibration of the stick. By providing a range finder
and/or proximity sensor on both ends of the stick, it is also
possible to inform the user not only of the surface of thestreet but also of obstacles at higher positions.
[0095] As an alternative to vibrator 44, it is possible to use
a pin array type display that can display uneven places
through haptic sensation. A pin array type display is covered
with many pins, and shapes are realized through haptic
sensation by moving the pins one by one up and down. In
this case, an actuator is used to move the pins. When moving
these many pins with a small device, a device used forBraille display can be used.
[0096] When a proximity sensor can be used to know the
shape of uneven places on the road, the shape can beconveyed to the user through this pin array type display.
[0097] Instead of vibrator 44, it is also possible to use an
inertial actuator. This actuator is provided with at least one
weight, and the weight is moved abruptly according to thedirection of the uneven place, thereby informing the user ofthe uneven place through the change in movement.
[0098] In this type of actuator, the weight is normally instandstill, and when displaying an uneven place, the weightis moved abruptly. If the weight is moved in the inside of arod-shaped tool, the weight can be moved forward andbackward, so two directional types of stimuli are possible.
As an alternative, it is possible to display two impact
directions by actually hitting the weight against the tip of thetool and the end of the tool.

[0099] The present invention is not limited to the above
embodiments; variations are possible within the scope of theclaims, which are incorporated in the scope of the present
invention.

US 2003/0057973 A1[0100] The component elements referred to herein do notalways mean physical means but include cases where thefunctions of each means are realized through software.
Furthermore, the functions of one means may be realized
through two or more physical means, or the functions of two
or more means may be realized through one physical means.
What is claimed is:

1. A smart tool for displaying non-haptic information
relating to the environment of a tool as haptic information to
the user of said tool, comprising:said tool;a sensor for detecting the environment surrounding said
tool;

an environmental information measurement portion for
gaining environmental information on said tool basedon the output from said sensor;
an environmental information/haptic sensation conversion portion for converting the gained environmental
information to haptic sensation; anda haptic display for conveying the converted haptic sensation to said tool.

2. A smart tool according to claim 1, wherein said sensor
is one of an optical sensor, an electrical sensor, a magnetic
field sensor, a temperature sensor, and a pressure sensor; said
environmental information is one of light-scattering intensity, color, distance to interface, electric resistance, electricfield intensity, magnetic field intensity, temperature, pressure, vibration, sound, and hardness; and said haptic sensa
tion is information sensed by the human being through senseof touch, and is one of a repulsive force working against theforce of the person operating the tool, a vibration of the tool
itself, heat generation of the tool, and an electric shock using
electric current.

3. A smart tool according to claim 1, wherein said sensor
is an optical sensor, said environmental information mea
surement portion measures the light-scattering intensity
based on the output from said optical sensor, and said
environmental information/haptic sensation conversion por
tion converts changes in said light-scattering intensity torepulsive force.
4. A smart tool according to claim 1, wherein said sensor
is an optical sensor, said environmental information mea
surement portion decides the color based on the output fromsaid optical sensor, and said environmental information/
haptic sensation conversion portion converts changes in saidcolor to repulsive force.
5. A smart tool according to claim 1, wherein said sensor

Mar. 27, 2003

including the roughness thereof, said environmental infor
mation measurement portion measures the surface state of
the object based on the output from said sensor, and said
environmental information/haptic sensation conversion por
tion converts changes in said surface state to vibration.
8. A smart tool according to claim 1, wherein said sensor
is a proximity sensor, said environmental information mea
surement portion detects an object in the vicinity of said tool
based on the output from said proximity sensor, and said
environmental information/haptic sensation conversion por
tion converts information on the existence of said object tovibration.

9. A smart tool according to claim 1, wherein said haptic
display is an actuator for pulling said tool.
10. A smart tool according to claim 1, wherein said haptic
display is a heater for heating said tool.
11. A smart tool according to claim 1, wherein said haptic
display is a source of current for applying an electric shockto the person holding said tool.
12. A smart tool according to claim 1, wherein said haptic
display is a vibrator for applying vibration to said tool.
13. A smart tool according to claim 1, wherein said haptic
display is a pin array type display for displaying unevenplaces through sense of touch.
14. A smart tool according to claim 1, wherein said haptic
display is an inertial actuator that comprises at least one
weight and that displays changes in the movement of said
weight through haptic sensation.
15. A smart tool according to claim 14, wherein said
actuator applies a force crossing the direction of the force
applied by a person to the tool, and applies a haptic sensation
to the person as if the tool itself were bending.
16. A smart tool according to claim 1, further comprising:a storage portion for pre-storing the optimal track along
which said tool should move, the speed and/or size ofthe force applied to said tool;a measurement portion for measuring the track of said
tool, the speed and/or size of the force applied to said
tool; anda processing portion for comparing the measurement
result of said measurement portion with the storage
contents of said storage portion, calculating the differ
ence between them, and converting the calculated dif
ference to haptic sensation.
17. A smart tool for displaying visual information in thevicinity of the tip of a tool as haptic information to the userof said tool, comprising:is a distance sensor, said environmental information mea

said tool;

surement portion measures the distance to an interface basedon the output from said distance sensor, and said environ
mental information/haptic sensation conversion portion converts changes in said distance to the interface to repulsive

a light-receiving element for receiving the light reflectedby an object in the vicinity of the tip of said tool;a decision portion for deciding the brightness and/or colorof the object in the vicinity of the tip of said tool basedon the output from said light-receiving element;a repulsive force calculation portion for calculating therepulsive force to be applied to said tool based on the
output from said decision portion; anda haptic display for applying repulsive force to said tool
based on the output from said repulsive force calculation portion.

force.

6. A smart tool according to claim 1, wherein said sensor
is an ammeter, said environmental information measurement

portion measures the electric resistance of the environment
based on the output from said ammeter, and said environ
mental information/haptic sensation conversion portion converts changes in said electric resistance to repulsive force.
7. A smart tool according to claim 1, wherein said sensor
is a sensor for measuring the surface state of an object

US 2003/0057973 A1

18. A smart tool for displaying information on the interface between liquids in the surrounding of a tool as haptic
information to the user of said tool, comprising:said tool;a first electrode attached to the tip of said tool;a second electrode placed within a liquid;
an ammeter for measuring the value of the electric current
flowing through said first electrode and said second
electrode;a decision portion for deciding the electric resistancebased on the electric current value measured by said
ammeter;a repulsive force calculation portion for calculating therepulsive force based on the gained electric resistance;
anda haptic display for applying repulsive force to said tool
based on the output from said repulsive force calculation portion.
19. A smart tool for displaying information on the interface between liquids in the surrounding of a tool as haptic
information to the user of said tool, comprising:said tool;a first electrode and a second electrode attached to the tipof said tool;

an ammeter for measuring the value of the electric current
flowing through said first electrode and said second
electrode;a decision portion for deciding the electric resistancebased on the electric current value measured by said
ammeter;a repulsive force calculation portion for calculating therepulsive force based on the gained electric resistance;
anda haptic display for applying repulsive force to said tool
based on the output from said repulsive force calculation portion.

Mar. 27, 2003

20. A smart tool for displaying information on the surface
of a distant object as haptic information to the user of a tool,comprising:a range finder for measuring the distance to said object;a distance change detection portion for detecting changes
in said distance;

a conversion portion for converting changes in thedetected distance to haptic sensation; anda haptic display for applying haptic sensation based on the
output from said conversion portion.
21. A smart tool for displaying information on the surface
of a distant object as haptic information to the user of a tool,comprising:a light-receiving portion for receiving the light reflectedby the surface of said object;a detection portion for detecting the state of said surface
based on the intensity of the light received by said
light-receiving portion;a conversion portion for converting the detected surface
state to haptic sensation; anda haptic display for applying haptic sensation based on the
output from said conversion portion.
22. A smart tool for displaying information on a distantobject as haptic information to the user of a tool, comprising:a proximity sensor for detecting information relating to an
object existing in the vicinity;a conversion portion for converting information on thedetected object to haptic sensation; anda haptic display for applying haptic sensation based on the
output from said conversion portion.
23. A haptic sensation generation method of a smart tool,comprising the steps of:an operator operating said tool;
measuring the environment in the surrounding of said tooland gaining environmental information other than hap
tic sensation;

calculating the haptic sensation to be applied to said tool
based on said environmental information; andapplying the calculated haptic sensation to said tool.