An Ungrounded Pen-shaped Kinesthetic Display:
Device Construction and Applications
Sho Kamuro
1)

1)

Kouta Minamizawa

In this paper, we implement an ungrounded pen-shapedkinesthetic display and construct a three-dimensional (3D)
haptic interaction system. The ungrounded pen-shapedkinesthetic display provides kinesthetic sensations to a user’s
fingers without the use of mechanical linkages. Therefore, theuser can move his/her hand freely in air and interact with
virtual environments with sensation of touching. We verified
the ability of the device to provide forces and to represent a
virtual surface for 3D input. Then we constructed a 3D haptic
interaction system in that the user can touch virtual objects
displayed as 3D images directly. We also applied the device toan interface for 3D modeling and constructed a 3D haptic
modeling system in that the user can create touchable 3D
models by sketching two-dimensional (2D) figures in air.
KEYWORDS: Haptic interface, kinesthetic display, haptic
interaction, 3D modeling.
INDEX TERMS: H.5.2 [Information Interfaces andPresentation]: User Interfaces—Haptic I/O; H.5.1 [Information
Interfaces and Presentation]: Multimedia Information
Systems—Artificial, augmented, and virtual realities
1

2)

The University of Tokyo

ABSTRACT

2)

Susumu Tachi

2)

Keio University

directly. On the other hand, wUbi-Pen [6], ImpAct[7], andSenstylus [8] are ungrounded pen-shaped interfaces and canprovide haptic sensations without limitations of hand
movements. However, the wUbi-Pen and ImpAct are usedwhile being touch with a screen surface and cannot be used for3D spatial interactions in air. Senstylus, on the other hand, is adual-rumble feedback device for 3D computer-aided-design(CAD) applications. The user can move their hand freely in air
and manipulate 3D objects by his/her movements. However, thedevice can provide only vibrations as haptic feedback, which
don’t represent a realistic sensation of touching a 3D model.
Although there are some ungrounded devices that are able torepresent kinesthetic sensations on the user’s hand [9, 10], theycannot provide continuous forces.
In our previous study [11], we proposed and prototyped an
ungrounded kinesthetic pen-shaped display. The device does
not restrict the movements of the user’s hand and can provide
kinesthetic sensations on the user’s fingers. In this paper, weconstruct a device that has three motors inside it (Figure 1) andenable haptic interactions with 3D virtual objects in a physicsbased simulation environment. We also evaluate theperformance of the device for 3D input and construct a 3Dmodeling system in that the user can create 3D shapes bysketching two-dimensional (2D) figures in air.

INTRODUCTION

In recent times, much research has been conducted on haptic
interfaces. By integrating with three-dimensional (3D) displays,
haptic interfaces realize users to touch 3D virtual objects andcan improve existence of virtual environments. Many haptic
interfaces have been proposed, and a pen-shaped haptic
interface is one of major types of them. To use a pen-shaped
device, a user is merely required to hold it and can interact with
virtual objects with sensation of pecking or tracing by the tip of
the pen. PHANToM [1] is a most popular pen-shaped haptic
interface. PHANToM enables a user to perceive kinesthetic
sensations on his/her hand with the help of mechanical linkages,
which are driven by multiple motors. This device has been
applied for many systems like 3D visual-haptic interaction
systems [2] or 3D modeling systems [3, 4, and 5]. However,there is one unavoidable problem in using PHANToM as aninterface; this device requires to be grounded to representkinesthetic sensations, and therefore it restricts movements of
the user’s hand within the range of the linkages. When using
grounded haptic interfaces like PHANToM, the user sees amonitor and move his/her hand at a position where the devices
are placed and the user cannot touch displayed virtual objects
7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan.
E-mail: kamuro@tachilab.org

Figure 1. Ungrounded pen-shaped kinesthetic display

2

UNGROUNDED KINESTHETIC PEN-SHAPED DISPLAY

In [11], we proposed a method to provide kinesthetic
sensations to the fingers of a user without a use of mechanicallinkages. To downsize the device, we hypothesized thatkinesthetic sensations on fingers alone are sufficient torepresent the sensations of touch. The proposed pen-shaped
device consists of following two parts; a grip at the position
where the pen is held by the user, and a base that is attached tothe user’s hand. The position where the base is fixed to thehand works as the point of support of force, and forces areprovided on the fingers by moving the grip toward the base. We

prototyped a pen-shaped device and confirmed that our
proposed method were able to provide kinesthetic sensations
and subjects could identify an increase in the provided forces.
This time, we designed and constructed an ungrounded penshaped kinesthetic display that has three built-in motors and aneasy-holding fixture. Figure 1 shows the constructed device
held by a user and Figure 2 shows the dimensional drawing of
the body of the device. The body of the device was cast byusing a rapid prototyping system (Dimension BST 768,
Stratasys, Inc.). Three motors (Maxon Motor Corp., RE 10,
1.5W, gear ratio = 1:16) are placed parallel to the central axis of
the pen. Strings are fixed to brass pulleys (6 mm across) using
screws, and these strings are tied at the connecting points of thegrip. The inner structure of the device is shown in Figure 3.
When the motors rotate and connecting points are pulled up, thegrip translates or leans toward the axis of the pen. A spring isplaced in the base and pushes back the grip when the motors are
not driven. The maximum force applied to the user using one
motor is 4.9 N. We also designed a holding fixture and an
openable and closable ring attached to the outside of the base sothat the user can start using the device with little effort. To use
the device, the user presses his/her index finger against the ringopen. Then, the ring is closed and the index finger is fixed tothe base.

3

DEVICE EVALUATION

3.1
Verification of Resolution to Provide Forces
To evaluate the representational ability of the constructed
device, we conducted an experiment in that subjects compared
two provided stimuli and answered which one was stronger.
This experiment was performed using a constant method.
The subjects included four males and one female (in their
twenties, right handed). In order to mask the sound of the
motors, white noise was presented to both ears of the subjects
via headphones. In this experiment, the subjects were instructedto hold the ungrounded pen-shaped kinesthetic display with
their right hand and close their eyes. In each trial, a standard
force and a comparing force were provided by the device inrandom order for one second at intervals of one second. Then,the subjects answered whether “the first force was stronger” or“the second force was stronger,” according to a two-alternative
forced-choice procedure.
In this experiment, forces were provided in two conditions asshown in Figure 4. In the first condition, all motors of thedevice were driven and the grip moved parallel to the central
axis of the pen. This motion generates the sensations of“pushing” or “pecking” virtual objects. In the second condition,
one motor was driven and the grip moved perpendicular to thecentral axis. This motion generates the sensation of friction or
the sensation of touching objects using the side of the pen.
In this experiment, the value of the standard force was set to
2.0 N, and seven values of the comparing force (0.8, 1.2, …,
3.2 N) were set. Twenty trials were performed for eachcondition of the value of the comparing force and the way
motors were driven; therefore, each subject performed a total of
one hundred and forty trials.

Figure 2. Dimensional drawing of the device.
The unit of measurement is mm.

Figure 4. Motion of the grip in each condition of experiment

Figure 3. Inner structure of the device.
The red line shows a string.

Figure 5 shows the rate of the responses that the comparing
force was stronger than the standard force. The curves represent
the fitted lines with a cumulative normal distribution. The pointof subjective equality (PSE) is defined as the value ofcomparison stimulus that produces a rate of the answer
corresponding to equiprobable random guessing (50 % for thisexperiment design). The difference between the PSE and the
75 % threshold is defined as the 75 % differential limen (DL).
When all motors were driven and the motion of the grip parallelto the central axis of the pen was generated, PSE was 2.0 N and
75 % DL was 0.40 N. When one motor was driven and the
motion perpendicular to the central axis was generated, on theother hand, PSE was 2.2 N and 75 % DL was 0.82 N.

Table 1 shows the average RMS (Root Mean Square) ofovershoots of the tip of the device from the surface of thevirtual canvas. The average RMS of all subjects was 52 mm in
the case with kinesthetic feedback, and was 127 mm in the casewithout kinesthetic feedback.

Figure 6. Prototype haptic interaction system

Figure 5. Result of force resolution experiment(Top: driving all motors, Bottom: driving one motor)

3.2
Evaluation of Application Possibility to 3D Input
We performed another experiment to evaluate the ability of
the constructed device to help 3D input by kinesthetic feedback.
By using a physics simulation engine, we constructed a virtualenvironment and presented a virtual canvas in air.
For this experiment, we constructed a prototype haptic
interaction system that enables the user to touch virtual objects
with the ungrounded pen-shaped kinesthetic display. Figure 5
shows an external appearance of the system. The size of thebooth is 90 x 90 [cm]. Six infrared cameras (OptiTrack FLEX:
V100, NatuarlPoint, Inc.) are placed at the top of the booth andthe position and the pose of the device are tracked by capturing
three markers fixed to the pen (see Figure 1). A 24-inch
monitor is placed and the user can see virtual objects placed
inside the booth. The virtual environment is constructed byusing a physics engine PhysX [12], and contacts between
objects and the tip of the kinesthetic display are simulated.
ContactReport class of PhysX calculates 3-DOF contact forces.
And based on the calculated forces, kinesthetic sensations are
represented on the user's fingers by driving the motors in thedevice. In this system, virtual objects and the pen-shaped proxythat represents the kinesthetic display are displayed on themonitor. The user can control the proxy by moving his/her handinside the booth, and touch virtual objects as if the displayed
objects exist in the real environment.
Figure 7 shows the setup for the experiment. For theexperiment, we placed a virtual canvas, which was with the sizeof 50 x 50 [cm] and was inclined toward the monitor at a 30degree angle, at the center of the booth. A circle, which was 30
cm in a diameter, was displayed on the face of the canvas, andthree subjects were instructed to trace it with the device in eachtrial. The subjects included two males and one female (in their
twenties, right handed). White noise was presented to both theears of the subjects via headphones, which masked the sound of
motors. Three trials were performed for each condition of withor without kinesthetic feedback. Therefore, each subject
performed six trials.

Position of virtual canvas

Figure 7. Setup for 3D input experiment
Table 1. Result of 3D input experiment [mm]without
Force
with
Force

subject1

subject2

subject3

average

±172

±140

±72

±127

±15

±88

±54

±52

3.3
Discussion
As shown in the results of the force resolution experiment, it
was confirmed that the constructed ungrounded pen-shapedkinesthetic display can apply multilevel forces to the fingers.
The device can apply forces of 14.7 N at the maximum in a
direction parallel to the central axis of the pen by driving all
motors. Therefore, the device has a high resolution and amaximum value to provide forces in this direction. However,
our proposed device provides kinesthetic sensations to fingers
by moving the grip and the range of movement of the grip has alimit. The maximum value of force that the user can perceive
must be measured as a future work. On the other hand, one or
two motors are used to provide forces perpendicular to thecentral axis, and therefore, the maximum value of the forces inthis direction is smaller. In addition, the results of theexperiment show that the resolution to provide forces in thedirection perpendicular to the central axis is lower and thedevice can apply force of levels with small numbers in thisdirection. There are differences in the effectiveness of force
provision among individuals because the fixation of the base of
the device to the hand is easily effected by how the user holds
the device. In the future work, we will design a holding fixture
that can be held more easily and firmly by different users.
However, the constructed device will work to some extent for
representing virtual objects with multilevel hardness andfriction.
In the 3D input experiment, contact forces between the tip of
the pen and the virtual canvas were provided. PhysX simulated
contacts between rigid bodies in the simulation environment
and the subjects could feel as if they were touching a rigid
canvas. In addition, 3-DOF forces were provided and thesubjects could recognize leanings of the canvas with kinesthetic
feedback. Figure 8 shows examples of the transitions ofovershoot in trials of the 3D input experiment with kinesthetic
feedback (shown as a red curve) and without kinesthetic
feedback (shown as a blue curve). This graph shows theeffectiveness of kinesthetic feedback with the ungrounded penshaped kinesthetic display to represent an input surface in 3D
manipulations. These results of this experiment encouraged us
to construct a 3D modeling system that supports the user’s
manipulations by representing a virtual input surface withkinesthetic feedback. With the use of the ungrounded penshaped kinesthetic display, intuitive input manipulations in air
will be achieved.

nVidia Corp.). We placed three markers on 3D Vision andtracked its position with OptiTrack cameras placed around thechassis, and displayed images appropriate to the user’s eyes.
Therefore, the user can see virtual objects from various
directions and touch them with the ungrounded pen-shapedkinesthetic display as though they were floating in air.
In this system, the user can touch displayed 3D images
directly with sensation of touching. This feature improves theexistence of virtual objects efficiently compared to theprototype system with 2D images. When the proxy that
represents the position of the tip of the device in the virtualenvironment is not displayed, the user can feel as though he/shewere touching objects that existed physically in the real world.

Figure 9. 3D haptic interaction system

5

Figure 8. Example of motion trajectories in 3D input experiment(performed by subject 3)

4

3D HAPTIC INTERACTION SYSTEM

The proposed device can provide kinesthetic sensations tothe user’s fingers without the use of mechanical linkages andthis advantage is useful to realize direct interactions with 3D
images. In this chapter, we construct a haptic interaction system(Figure 9) in that the user can touch virtual objects by using theconstructed kinesthetic display. We display the virtualenvironment with 3D image display. Hence, the user can see
virtual objects as 3D images and touch or move them withkinesthetic feedback.
We constructed the haptic interaction system based on the
prototype system constructed in section 3.2. The virtualenvironment is constructed using PhysX, and contact forces arecalculated for kinesthetic feedback. The position and the poseof the device are tracked by OptiTrack cameras, and the motionof the user’s hand is reflected to the virtual environment.
To display the virtual environments with 3D image, we
implemented a chassis with a 40-inch screen and projected
images by a projector placed inside the chassis. The user cansee 3D images on the screen by wearing 3D glasses (3D Vision,

3D HAPTIC MODELING SYSTEM

The haptic interaction system constructed in chapter 4
encouraged us to apply the ungrounded kinesthetic display to asystem in that the user can create and manipulate 3D objects
directly in air. In this chapter, we construct a 3D haptic
modeling system by integrating the haptic interaction systemwith a 3D modeling software. To enable an easy and intuitive
3D modeling, the manipulation that the user has to perform tocreate 3D shape should be as simple as possible. Therefore,
based on the result of the experiment in section 3.2 and themodeling method of Teddy [13], we realized a 3D shapecreation by drawing 2D figures in air.
We support the user to draw figures in mid-air by reproducing
the virtual canvas as done in the 3D input experiment. We placed
an input button at the grip of the ungrounded pen-shapedkinesthetic display and enabled to switch operations by pushing or
releasing the button. When the user pushes the button for the firsttime, the virtual canvas appears at the tip of the pen. The positionof the device is tracked by using OptiTrack cameras, and the user
can place the virtual canvas at a desired position by moving
his/her hand. The virtual canvas is generated in the physics-based
simulation environment and displayed as a 3D image. When theuser touches the canvas, the ungrounded pen-shaped kinesthetic
display generates reaction forces calculated by PhysX based on
the degree of the contacts. Therefore, the user can draw 2D
figures on the virtual canvas as if he/she were drawing on a realcanvas with physical existence.

We implemented an operating panel for 3D modelling (Figure10) and placed it in front of the chassis. To create a new shape, theuser selects the desired shape from among a sphere, a box, andfreeform surfaces. Then the user sketches a 2D figure of thedesired shape on the virtual canvas, for example, a circle, a
rectangle, or a free closed curve. The motion trajectory of the tipof the pen is recorded while the input button is pushed and held.
Based on the tracked trajectories, parameters of the shape arecalculated and the desired shape is created in the environment.
Figure 11 shows how 3D shapes are created from motion
trajectories. By these implementations, touchable 3D shapes weresuccessfully created by an easy operation of sketching 2D figuresin air.
We also implemented sketching in air without the virtual
canvas in addition to sketching with canvas described above. In
this mode, the user can sketch figures directly in the air byholding the input button and moving his/her hand. A constantforce is provided to the user’s fingers while the button is pushed
and held as if the user were sketching figures on an invisible
canvas. The tracked motion trajectories are fit to a surface with aleast square algorithm and parameters of the desired shapes arecalculated.

Figure 10.

and friction of the shape. Then the user draws 2D figures on thevirtual canvas or in air, which are converted to 3D shapes. By
combining multiple shapes, the user can create complex 3Dmodels easily. The created 3D models can be seen from various
directions and be touched by the ungrounded pen-shapedkinesthetic display. The values of the hardness and friction of the
shapes are reflected to forces calculated by PhsyX as factors. The
user can confirm shapes and haptic impressions of the models orinteract with them. Figure 13 shows an example of 3D models
created by using this system.
In the prototype haptic interaction system constructed in
section 3.2, the 3D virtual environment was displayed on the 2D
monitor. Therefore, the subjects of the experiment were not able
to estimate the position and the pose of the canvas placed in thebooth accurately without kinesthetic feedback provided by theungrounded pen-shaped kinesthetic display. We constructed thehaptic modeling system with kinesthetic feedback and 3D image
display. This integration realized the intuitive manipulation as if
the user were drawing a canvas that existed in the real world. The
representation of the virtual canvas is effective not only for
intuitive input manipulation but also accurate model creations; theuser can coordinate the position of shapes where they will be
generated by moving the canvas. This position coordination
method is more intuitive than methods used in conventionalmodeling systems and the user can design 3D structures of themodels easily by using the system.

Operating panel

Figure 12.

Figure 11.

3D haptic modeling system

Creations of shapes from motion trajectories(Top: Box, Bottom: Shape with freeform surfaces)

Figure 12 shows the constructed 3D haptic modeling system.
The user wears 3D glasses, holds the ungrounded pen-shapedkinesthetic display by his/her right hand, and stands ahead of thechassis. Then, the user touches the operating panel to select thedesired operations. The user can also configure the color, hardness,

Figure 13.

Example of 3D models
created by using the system

6

CONCLUSION

In this paper, we implemented an ungrounded pen-shapedkinesthetic display and constructed two haptic-enabled systems; a
3D haptic interaction system and a 3D haptic modeling system.
Based on the previously-proposed method, we implemented thedevice that provides kinesthetic sensations to fingers and verified
the force resolution and the application possibility to 3D input ofthe device. Then we constructed a system in that a user can see
and touch 3D virtual objects with sensation of touching. We also
applied the device to 3D modeling, and constructed a system thatenables the user to create touchable 3D shapes by sketching 2D
figures in air with kinesthetic feedback.
As the proposed device does not impose restrictions on theuser’s movements, it can be used in more space than conventional
kinesthetic displays. And also, multiple users can use the devices
respectively at same time in same place because each device
works without interferences. Taking advantage of these features,
we are planning to construct a system in that multiple users canenjoy or work together in virtual environments. The constructed
3D haptic interaction system that improves existence of virtual
objects encouraged us to develop some interactive applications forcommunicating with virtual creatures. Additionally, more intuitive
3D modeling methods can be proposed as future works. In theconstructed 3D haptic modeling system, the user can only createshapes and turn off a 3D model by putting them together. We
intend to implement transforming manipulations on shapes bypushing or scarping with kinesthetic feedback. We will also
implement alternative methods to create shapes by sketching, forexample, combining two figures to create oval sphere or complex
box shapes or indicating a surface of the shape by drawing spirals.
ACKNOWLEDGEMENTS
This work has been supported by CREST of JST (Japan
Science and Technology Agency).
REFERENCES
[1]

[2]

[3]
[4]

[5]
[6]

[7]

[8]

T. Massie and K. Salisbury, “The PHANToM Haptic Interaface: A
Device for Probing Virtual Objects,” In Proceedings of the ASME
Winter Annual Meetings, Symposium on Haptic Interfaces for
Virtual Environment and Teleoperation Systems, 1994.
J. D. Brederson, M. Ikits, C. R. Johnson, and C. D. Hansen, “TheVisual Haptic Workbench,” In Proceedings of 5th PHANToM User
Group Workshop (PUG), 2000.
SensAble Technologies, Inc., FreeForm Systems,
http://www.sensable.com/products-freeform-systems.htm
A. D. Gregory, S. A. Ehman, and M. C. Lin, “inTouch: Interactive
Multiresolution Modeling and 3D Painting with a Haptic Interface,”
In Proceedings of the IEEE Virtual Reality 2000 Conference, pp. 4549, 2000.
M. Foskey, M. A. Otaduy, and M. C. Lin, “ArtNova: touch-enabled
3D model design,” ACM SIGGRAPH 2005 Courses, 2005.
KU Kyung, and JY Lee, “wUbi-Pen: windows graphical userinterface interacting with haptic feedback stylus,” ACM SIGGRAPH
2008 New Tech Demos, 2008.
A. Withana, M. Kondo, G. Kakehi, Y. Makino, M. Sugimoto, and M.
Inami, “ImpAct: Immersive Haptic Stylus to Enable Direct Touchand Manipulation for Surface Computing,” 7th International
Conference on Advances in Computer Entertainment Technology(ACE 2010), Taiwan, November 17-19, 2010.
M. Fiorentina, G. Monno, and A. Uva, “The Senstylus: a novel
rumble-feedback pen device for CAD application in virtual reality,”
In Proceedings of the 13th International Conference in Central
Europe on Computer Graphics, Visualization and Computer Vision‘2005 (WSXG 2005), 2005.

[9]

T. Amemiya, H. Ando, and T. Maeda, ”Virtual Force Display:
Direction Guidance Using Asymmetric Acceleration via Periodic
Translational Motion,” In Proceedings of World Haptics Conference,
pp. 619-622, 2005.
[10] M. Sakai, Y. Fukui, and N. Nakamura, “Effective Output Patterns
for Torque Display 'GyroCube,' ” In Online Proceedings of the 13th
International Conference on Artificial Reality and Telexistence,
2003.
[11] S. Kamuro, K. Minamizawa, N. Kawakami, and S. Tachi.
“Ungrounded Kinesthetic Pen for Haptic Interaction with Virtual
Environments,“ In Proceedings of IEEE International Symposium on
Robot and Human Interactive Communication (IEEE RO-MAN
2009), pages 436–441, 2009.
[12] nVidia Corporation., PhysX, http://www.nvidia.com[13] T. Igarashi, S. Matsuoka, and M. Tanaka, “Teddy: a sketchinginterface for 3D freeform design,” In ACM SIGGRAPH’99, pp. 409416, 1999.