The 18th IEEE International Symposium on
Robot and Human Interactive Communication
Toyama, Japan, Sept. 27-Oct. 2, 2009

WeIAH.2

Ungrounded Kinesthetic Pen for Haptic Interaction
with Virtual Environments
Sho KAMURO, Kouta MINAMIZAWA, Naoki KAWAKAMI, and Susumu TACHI

Abstract—We proposed a pen-shaped handheld haptic
display that provides kinesthetic sensations to the fingers of auser without the use of mechanical linkages. The user’s
movements are not restricted since the device does not havemechanical linkages, and the user can enjoy haptic interactions
with virtual environments. In order to downsize the device, we
designed our device on the basis of a hypothesis that kinesthetic
sensations on the user’s fingers alone are sufficient to represent
the sensations of touch. We implemented a prototype device
and performed an experiment to confirm the representational
ability of our device. We also developed a prototype haptic
augmented reality system, using which the user can see andtouch a computer graphics object.

V

I. INTRODUCTION

ARIOUS haptic displays that provide the sensations oftouch to enable interactions with virtual objects havebeen developed. However, simple and user-friendly haptic
displays that produce realistic sensations of touch and can beeasily applied to conventional virtual reality systems for
haptic augmentation still need to be developed.
CyberGrasp [1] is a typical wearable haptic display,which is worn by a user, and it provides kinesthetic
sensations to the fingers of a user. The user feels as if he/she
is directly touching virtual objects with his/her hand.
However, most wearable haptic displays are cumbersome
and their use requires extra effort since users have to wear
them. Handheld haptic displays are another type of haptic
displays, which are more easy to use than wearable haptic
displays, because the user needs to merely grasp the device
to start using it. PHANTOM [2] is a typical pen-shaped
handheld haptic display, which provides kinesthetic
sensations to the hand of a user who is grasping the
pen-shaped interface. The sensations provided by
PHANTOM represent intimate kinesthetic sensations,which are provided with the help of mechanical linkages
driven by multiple motors. However, this device must be
grounded so that it restricts the user’s movements to a rangeof mechanical linkages. Recently developed portablehandheld haptic displays such as wUbi-Pen [3] andSenstylus [4] are capable of providing haptic sensations
without the use of mechanical linkages. Although such
Sho Kamuro, Kouta Minamizawa, Naoki Kawakami are with the
University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8656, Japan.
Susumu Tachi is with Keio University, 4-1-1 Hiyoshi, Minatokita-ku,
Yokohama-shi, Kanagawa, 223-8526, Japan. Tel: +81-3-5841-6917, Fax:
+81-3-5841-8601. E-mail: sho_kamuro@ipc.i.u-tokyo.ac.jp

978-1-4244-5081-7/09/$26.00 ©2009 IEEE

ungrounded devices do not impose restrictions on the user’s
movements, there are other problems associated with the useof these devices; in order to effectively use wUbi-Pen, it isessential to maintain a physical contact between the device
and the screen surface, and this device does not function if it
is used in mid-air; on the other hand, Senstylus can be usedin mid-air; however, it can only produce vibrations, which
do not satisfactorily represent the realistic experience of
touching objects. Although there are a number of
ungrounded devices that provide kinesthetic sensations on
the basis of a characteristic feature of human beings [5],these devices are capable of providing only periodic
kinesthetic sensations. Moreover, the development of an
ungrounded haptic display that can provide continual
kinesthetic sensations has not been reported thus far.
In this study, we propose a method to provide kinesthetic
sensations to the fingers of a user without the use of
mechanical linkages and develop a pen-shaped haptic
display using this method (Fig. 1). The results of ourexperiment using a prototype device show that our proposed
method can successfully provide kinesthetic sensations. We
also develop a prototype haptic augmented reality systemusing this device.

Fig. 1. Conceptual drawing of our proposed device

II. PROPOSED METHOD
Human haptic sensation includes cutaneous sensation of
the skin and kinesthetic sensation produced at the joints ofthe fingers and arms. As mentioned in the previous chapter,conventional portable haptic displays, which do not restrict
the user’s movements, provide only cutaneous sensations or
periodic kinesthetic sensations. In general, it is essential touse a large device to provide kinesthetic sensations to theuser’s arms; however, such large devices are seldom
portable. On the other hand, it is possible that providing
continual kinesthetic sensations only to the fingers will
represent better sensations of touch than cutaneous436

sensations or periodic kinesthetic sensations.
Therefore, we propose a haptic display on the basis of ahypothesis that kinesthetic sensations on the user’s fingers
alone are sufficient to represent the sensations of touch. Thishypothesis is efficient because we don’t need to provide
kinesthetic sensations to the arms; therefore the device canbe downsized. Fig. 1 shows a conceptual drawing of our
proposed device. The device is pen-shaped, which is easy to
grasp. To apply a force using an ungrounded device, it isessential to place the point of support of force on some partof the user’s body so as to set the sum of internal forces tozero. Therefore, we have to design our portable haptic
display in such way that the point of support and the point ofapplication of force are enclosed within the hand itself. We
fixed the supporting point of force as a point on the base of
the index finger, and we applied force to the fingertips of theindex finger, middle finger, and thumb by changing thelength of the pen-shaped device. Fig. 2 shows the schematicof the mechanism of our proposed device. The device
consists of the following two parts: a part from where thepen is held (grip part) and a base part that is fixed to the hand.
When a user grasps this device, the base part is fixed to thebase of the user’s index finger, which is inserted in the ringattached to the outside of the base part. The user grasps thegrip part by the tips of his/her index finger, middle finger,
and thumb. When the grip part moves toward the base partwith the help of multiple motors in the base part, kinesthetic
sensations are provided to the three fingers that grasp thegrip part. Inside the base part, three motors and strings are
fixed, which pull each connecting point in the grip part andcontrol its 3-DOF motion, as shown in Fig. 2. When all thethree motors wind the strings and pull each connecting point,
the grip part moves parallel to the central axis of the pen as itis pulled into the base part. On the other hand, when only
one or two motors are driven, the grip part moves
perpendicular to the central axis as it is moved away fromthe axis. The motion parallel to the central axis generates thesensations of “pushing” or “pecking” virtual objects, and the
motion perpendicular to the central axis generates thesensation of friction or the sensation of touching an object
using the side of the pen.

III. PROTOTYPE DEVICE
To confirm the availability of our proposed method, wedeveloped a prototype device. Fig. 3 shows the outerappearance of the device, and Fig. 4 shows the dimensional
drawing of the inner structure of the device. As mentioned
above, our device is composed of the base part, which isfixed to the hand, and the grip part, which moves toward thebase part. The body of the device was cast using a rapid
prototyping system (Dimension BST 768, Stratasys Inc.). As
shown in Fig.2, although the motors are placed inside thebase part parallel to the central axis of the pen, we placed
three motors (Maxon Motor Corp., RE 10, 1.5 W, gear ratio= 1:16) at the bottom of the base part perpendicular to thecentral axis in order to simplify the implementation of thedevice. Strings are fixed to brass pulleys (6 mm across)using screws. These strings are laced through holes present
in the base part, and they are tied at the connecting points in
the grip part. The 3-DOF motion of the grip part iscontrolled by pulling up the connection points using the
motors. The maximum force applied to the user using one
motor is 4.9 N. A spring is also placed in the base part,which pushes back the grip part when there are no input
powers to the motors.
We also set up a holding fixture outside the base partusing paper clay that covers the base of the user’s thumb andindex finger so that the user can hold the device firmly. The
ring for the fixation of the hand and the base part is attachedoutside the base part. Fig. 5 shows the prototype device thatis held by user in his/her right hand.

Fig. 2. Schematic of the mechanism of our proposed device
Fig. 3. Implemented prototype device437

to the tip of the device, and the position of the device wasmeasured by capturing the IR LED with a Bluetooth IR
camera (WiiRemote, Nintendo Co., Ltd.). Position sensing
was performed at a resolution of 0.3 mm and a frequency of
200 Hz. The virtual wall and the position of the tip of thedevice were drawn on the display of a laptop, and the subject
could touch the virtual wall while looking at the display.
When the position of the tip of the pen is expressed as x(mm) and the position of the edge of the virtual wall isexpressed as x0 (mm), the force F (N) applied by the device
is calculated using the following equation. In this equation, k(N/mm) is a factor that corresponds to the hardness of thevirtual wall, and F0 (N) is the maximum force applied that
does not move the grip part.

( x < x0 )0⎧F =⎨⎩ k ( x − x 0 ) + F 0 ( x ≥ x0 )Fig. 4. Dimensional drawing of prototype device.
The unit of measurement is mm.

Fig. 5. Prototype device grasped by hand

IV. EXPERIMENTS
A. Hardness Detection Experiment
In order to confirm whether kinesthetic sensations can beprovided by our proposed method and validate therepresentational ability of our prototype device, we
performed an experiment to measure the difference limen(DL) of the hardness detected using the prototype device.
In this experiment, the subject touched two virtual walls
using the prototype device and mentioned which wall was
harder. Fig. 6 shows the experimental setup, and Fig. 7
shows the coordinate system used in the experiment.
Since the position of the center of gravity moved toward
the bottom of the device due to the motors placed at thebottom of the pen, we cancelled the gravitational bias bysuspending a weight through a sheave with a string attachedto the bottom of the base part. The weight was 70 g, which
was nearly equal to the total weight of the three motors andpulleys. We also placed a double-sided tape on the grip partin order to prevent slipping between the grip part and theuser’s finger tips. An infrared LED (IR LED) was attached(1)

In this experiment, seven values of k (0.3, 0.4,..., 0.9) wereset, hereinafter called ki (i = 0, 1,..., 6), and the value of F0
was set to 0.5 N.
This experiment was performed using a constant method.
The subjects included two males and one female (in their
twenties, right handed). White noise was presented to boththe ears of the subjects via headphones, which masked thesound of motors. The subjects were instructed to sit on achair and hold the prototype device with their right hand. In
each trial, two walls (the standard wall and the comparingwall) were presented in random order. The hardness of thestandard wall was k3 = 0.6 N/mm, and that of the comparing
wall was ki (i = 0, 1,..., 6). The subjects touched both thesewalls sequentially by moving their right hands along thex-axis, as shown in Fig. 7. Then, the subjects answered
whether “the first wall was harder” or “the second wall was
harder”, according to a two-alternative forced-choiceprocedure. Twenty trials were performed for each conditionof k; therefore, a total of one hundred and forty trials were
performed by each subject.

438

Fig. 6. Experimental setup

motor was 4.0 N. The movement of the tip part wasmeasured by capturing a marker attached to the tip of thedevice with an optical tracking system with multiple
cameras (NaturalPoint Inc., OptiTrack FLEX: V100).
Position sensing was performed at a resolution of 1 mm anda frequency of 100 Hz.
In order to stretch the strings, the minimum output force
was set to 0.5 N in each trial, which was the maximum force
that did not move the grip part. After 250 ms, the output
force was set to 4.0 N for 500 ms. Then the output force was
set to 0.5 N again for 750 ms. We performed this trial fivetimes.
Fig. 9 shows the target force and the measured force
calculated from the average migration distance of the tip of
the pen at each time step. The gap between the rise of input
and the time the output force reached the maximum value
was about 100 ms in average.

Fig. 7. Configuration of hardness detection experimentFig. 8 shows the rate of the responses that the comparing
wall was harder than the standard wall. The curves represent
the fitted lines with a cumulative normal distribution. The
point of subjective equality (PSE) was 0.63 N/mm, which
was nearly equal to the value of k of the standard wall (0.6N/mm), and 75 % DL was 0.12 N/mm, which was derived
from the difference between the PSE and the 75%
discrimination threshold.

Fig. 9. Result of system delay experimentFig. 8. Result of hardness detection experiment

B. Device Delay Experiment
We performed another experiment in order to estimate a
responsivity of the prototype device. We estimated the delayof the device by measuring the movement of the tip part ofthe device when rectangular-wave currents were input tothree motors of the device.
In this experiment, the base part was fixed so that thecentral axis of the pen became vertical to the ground. The tip
part of the device could move parallel to the central axis of
the pen by about 7 mm, and the spring constant of the springused in the prototype device was 0.57 N/mm. Therefore, theoutput force we could estimate by measuring the migrationof the tip part was 4.0 N at maximum. We set the maximum
amplitude of the input current as the force presented by the

C. Discussion
As shown in Fig. 8, it was confirmed that kinesthetic
sensations were provided using our proposed method, andthe subjects could identify an increase in the force by theovershoot of the tip of the pen from the edge of the virtual
wall. The 75% DL for standard hardness (0.6 N/mm) was
0.12 N/mm. The maximum force applied by the prototype
device was 14.7 N; when a subject touched a virtual wall andthe overshoot reached a certain value, a force of 14.7 N was
applied by the device. Moreover, this force remained
constant even if the overshoot increased beyond a certain
value. In view of these facts, the resolution of the force
applied by the device can be calculated by assuming that the
subjects compared the hardness of two walls on the basis of
the forces provided from these walls in the same overshoot.
By comparing the standard wall with the wall that had k of
0.48 N/mm, which was calculated by subtracting the 75%
DL from the standard hardness, it was found that thedifference between the two forces applied to these walls
reached a maximum value of 2.84 N with an overshoot of
23.7 mm. By comparing the standard wall with the wall that439

had k of 0.72 N/mm, which was calculated by adding the
75% DL to the standard hardness, it was found that thedifference between the two forces reached a maximum valueof 2.4 N with an overshoot of 19.7 mm. In this experiment,
since the subjects could successfully discriminate between
the two walls, the maximum resolution of the force appliedby the device was 2.4 N. Therefore, our prototype device iscapable of applying force of at least seven levels. However,
the responses of the subjects are greatly influenced by manyfactors (for example, how each subject touches the walls or
how they arrive at a particular answer). Therefore, for a
detailed verification of resolution, it is essential to perform
an experiment under detailed conditions.
In the device delay experiment, an oscillating motion of
the tip part around t = 900 ms occurred because of theinfluence of the spring. Though this movement is cranky, it
would not be recognized in practical use because the tip partis grasped by fingers.
The latency of the device was about 100 ms in the device
delay experiment. The device is thought to be worth
practical use enough with this value of latency. As our
method provide the kinesthetic sensations by moving the tip
part of the device by motors, it takes some times for the
output force to reach at a maximum value as motors need toroll up strings some distance (7 mm in the prototype device).
However, on practical use, the tip part is grasped by theuser’s fingers and the tip part does not move in ideal
condition. Therefore, the latency of the device will reduce
when it is used in the practical system. To estimate the delayof the force presentation in the system, we will measure theforce presented on fingers of the user and verify the delay of
the force presentation on practical use. There was also a gap
between the start of input and the start of the tip part's
movement, which was about 20 ms in average. This delay is
supposed to be arisen from frictions in the device. We
should implement the device while taking the frictions
between moving parts in the device into consideration.
V. APPLICATION
Our proposed device can provide intimate kinesthetic
sensations without using mechanical linkages; therefore, auser can use the device in mid-air without any restriction on
his/her movement. Because of such simplicity and usability,
the device can be easily applied to various types of
conventional virtual reality systems that are not hapticallyaugmented.
We implemented a haptic augmented reality system (Fig.
10) as an example of such an application. Using this system,
the user can touch a virtual cube that is drawn additionallyover the image of the real world by using an ARToolKit [6].
The computer graphics cube was drawn on the ARToolKit
marker, which was captured by a web camera. Kinesthetic
sensations were provided by our proposed device whose tip

was in contact with the cube. We attached a retroreflective
marker instead of an IR LED to the tip of the pen because it
is possible that the power strip for the IR LED might hide theARToolKit marker. Furthermore, we captured the positionof the tip of the device using an IR camera. Using the same
approach as that used in the experiment, the applied force
was calculated using equation (1). Therefore, the user could
see the virtual cube augmented on the real image on thescreen and also touch the cube using our proposed device.

Fig. 10. Haptic augmented reality system

VI. CONCLUSION
In this study, we proposed a method to provide kinesthetic
sensations to the fingers of a user using a portable device
without the use of mechanical linkages. Using this method,
we implemented a prototype pen-shaped haptic display andshowed that our device can apply force of at least sevenlevels. We also estimated the delay of the device andconfirmed that our device is worth practical use. Then, wedeveloped an augmented reality system using which the user
could see and touch a computer graphics cube in the realworld.
We have tested only the force that is applied parallel to theaxis of the pen. We intend to perform an experiment to test
the force that is perpendicular to the axis of the pen andshow the effectiveness of our device in representing realistic
sensations of touching virtual objects. Taking advantage ofeasy accessibility and good representational ability of our
proposed device, we also intend to develop more interactive
applications for multiple users.
ACKNOWLEDGMENT
This work was partially supported by CREST of JST(Japan Science and Technology Corporation).
REFERENCES
[1]
[2]

440

Immersion Co., CyberGrasp: Groundbreaking haptic interface for theentire hand, http://www.immersion.com/3d/products/.
SensAble Technologies, Inc., PHANTOM,
http://www.sensable.com/products-haptic-devices.htm.

[3]KU Kyung, and JY Lee, “wUbi-Pen: windows graphical user interface
interacting with haptic feedback stylus,” ACM SIGGRAPH 2008 New
Tech Demos.
[4] M Fiorentino, G Monno, and A Uva, “The Senstylus: a novel
rumble-feedback pen device for CAD application in virtual reality,” In
Proceedings of the 13th International Conference in Central Europe
on Computer Graphics,Visualization and Computer Vision'2005(WSCG 2005), 2005.
[5] T Amemiya, I Kawabuchi, H Ando, T Maeda, “Double-layer
slider-crank mechanism to generate pulling or pushing sensation
without an external ground,” In Proceedings of IEEE/RSJ
International Conference on Intelligent Robots and Systems, pp.
2101–2106, 2007.
[6] H. Kato and M. Billinghurst, “Marker tracking and HMD calibration
for a video-based augmented reality conferencing system,” In
Proceedings of IWAR '99, pp. 85-94, 1999.

441