3D Haptic Modeling System using
Ungrounded Pen-shaped Kinesthetic Display
Sho Kamuro
1)

1)

Kouta Minamizawa

The University of Tokyo2)

2)

Susumu Tachi

2)

Keio University

ABSTRACT
We propose a three-dimensional (3D) haptic modeling system thatenables a user to create 3D models as though he/she is drawing
pictures in mid-air and touch the created models. In the proposedsystem, we use our ungrounded pen-shaped kinesthetic display asthe interface. The kinesthetic display device can generatekinesthetic sensations on the user's fingers. Because our systemdoes not use mechanical linkages, the user can freely move his/herhand and feel the sensation of touching virtual objects. In theproposed system, the user can easily and intuitively create various
3D shapes by drawing closed curves in air using the device. The
created shapes are generated in a physics-based simulation
environment and are displayed as 3D images. Therefore, the user
can touch and see the drawn shapes as though they exist in reality.
KEYWORDS: Haptic interface, kinesthetic display, 3D modeling.
INDEX TERMS: H.5.2 [Information Interfaces and Presentation]:
User Interfaces—Haptic I/O; H.5.1 [Information Interfaces andPresentation]: Multimedia Information Systems—Artificial,augmented, and virtual realities
1INTRODUCTION

In recent times, three-dimensional (3D) modeling methods havebeen researched extensively. Traditional 3D modeling systems aredifficult to use, and only skilled users are able to create complex
models. However, several intuitive 3D modeling methods thatenable general users to create complex 3D models have beendeveloped. For example, Igarashi [1] proposed a 3D sketchinginterface named Teddy. Teddy allows its user to create freeform
surfaces and 3D models by sketching two-dimensional (2D)
strokes using a mouse or a pen-tablet system. Another study
related to 3D modeling using 2D strokes was conducted by Rivers
er al. [2]. In this study, multiple 2D vector art drawings of a
cartoon character were used to construct a two-and-a-halfdimensional (2.5D) model. The constructed model was used tosimulate 3D rotations and generate reasonable renderings of thecartoon from any angle. Using the techniquies employed in theabovementioned studies, users can create various 3D models
without prior knowledge of 3D modeling.
The methods used in the abovementioned studies wereproposed to realize intuitive 3D modeling with 2D input devices.
Another approach is to use 3D input devices such as hapticinterfaces. By using haptic interfaces in 3D modeling systems, theuser can treat virtual 3D models as though he/she were touchingor manipulating the model in reality. FreeForm [3] is a 3D
7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan.
E-mail: kamuro@tachilab.org

Figure 1. 3D haptic modeling system. A user can see a created 3Dmodel thorough 3D glasses, and touch the generated modelwith an ungrounded pen-shaped kinesthetic display.

modeling system used in conjunction with a popular pen-shaped
haptic display PHANToM [4]. Using PHANToM, the user cancreate 3D models by shaping virtual clays as if he/she wereactually touching the clays displayed in the monitor. Some studies[5, 6] have also been conducted on 3D modeling systems that use
PHAToM as the interfaces and allow users to create 3D models
aided by visual and haptic feedback. Although theabovementioned systems consider actual 3D movements of theuser’s hand as input, some unavoidable problems are caused by
the use of PHANToM. PHANToM has mechanical linkages that
represent intimate kinesthetic sensations and must be grounded.
Therefore, it restricts the user to a limited range of movements.
SenStylus [7] is a pen-shaped device that provides dual-rumble
feedback. When using this device, the user can move his/her handfreely, and such spatial movements are used as input. Although
SenStylus does not restrict the user’s movements, it can produce
only vibrations as feedback, which do not generate a realistic
sensation of touching a 3D model.
In our previous study [8], we proposed and implemented an
ungrounded pen-shaped kinesthetic display. Although our device
is ungrounded and portable, it can generate kinesthetic sensations
on a user’s fingers. The device consists of two parts; a grip at heposition where the pen is held by the user, and a base that isattached to the user’s hand. Three motors inside the base part
control the three-degree-of-freedom motion of the grip, and thismotion generates forces on the user’s fingers. In this poster, wedescribe the construction of a haptic 3D modeling system thatenables the user to create 3D models as if he/she were drawing
them in air (Figure 1). The use of our ungrounded pen-shapedkinesthetic display as the system interface allows this system tosimultaneously support a wide-range input area and provide
realistic haptic feedback for intuitive object manipulations.

2

SYSTEM OVERVIEW

Figure 1 shows the constructed 3D haptic modeling system. Theuser wears 3D glasses (3D Vision, nVidia Corp.), holds an
ungrounded pen-shaped kinesthetic display in his/her hand, andstands in front of the chassis.
In the proposed system, the introduction of an easy andintuitive modeling method is desired, and the manipulation thatthe user has to perform to create a 3D shape should be as simple
as possible. To this end, we developed a system that allows users
to create 3D models by sketching 2D figures in air. The user candraw figures in air on a virtual canvas reproduced by the system.
The grip of the ungrounded pen-shaped kinesthetic display has abutton that can be pushed and released to switch operations. Whenthe user pushes the button for the first time, the virtual canvas
appears at the tip of the pen. Motion tracking cameras (OptiTrack,NaturalPoint, Inc.) track the device position, and the user canplace the virtual canvas at a desired position by moving his/herhand. The virtual canvas is generated in a physics-based
simulation environment constructed using PhysX (nVidia Corp.)
and is displayed as a 3D image. When a user touches the canvas,
the kinesthetic display device generates reaction forces based on
the degree of contact. Therefore, the user can draw images on thevirtual canvas as if he/she were drawing on a real canvas.
To create a new shape, the user selects the desired shape fromamong a sphere, a box, and freeform surfaces. Then the user
sketches a 2D figure of the desired shape on the virtual canvas, forexample, a circle, rectangle, or free closed curve. The motion
trajectory of the tip of the pen is recorded for the time that theinput button is pushed and held. Based on the tracked trajectory,
parameters of the shape are calculated and the desired shape iscreated in the physics-based simulation environment (Figure 2).
Thus, 3D shapes were successfully created by sketching 2D
figures in air.

now. However, such content will be necessary in the future
because of the predicted widespread use of 3D televisions andgaming systems that use the user’s body movements as input. Our
system will serve as a basis for the creation of such haptic
contents.
3ACKNOWLEDGEMENTS
This work was supported by CREST of the Japan Science andTechnology Agency.
REFERENCES
[1]

[2]
[3]
[4]

[5]

[6]
[7]

Figure 2. Creation of shapes from motion trajectory(Top: Box, Bottom: Shape with freeform surfaces)

The created 3D shapes are generated in the physics-based
simulation environment and displayed on the screen as 3D images.
The user can see the created shape through 3D glasses and touch itin real time using the kinesthetic display. Therefore, the user feels
as though he/she is creating 3D shapes in reality.
Further, using our system, the user can touch the created 3Dobjects and feel their shape. This feature is beneficial not only inchecking the modeling process, but also in demonstrating thecreated models to other users. Haptic 3D models are not popular

CONCLUSION

In this poster, we have described the construction of a 3D haptic
modeling system to easily and intuitively create 3D models. We
propose an input method wherein the user draws 2D figures on avirtual canvas to create 3D shapes. The constructed system
realizes 3D modeling by creating and combining shapes on thebasis of the spatial movements of the user’s hand, and enables
real-time haptic interactions with the created 3D models.
Although the input-related manipulations in our system are
simple and the user can quickly create 3D models, otherapproaches for model creation with 3D input will be developed.
An alternative method is to use multiple 2D figures to create one
3D shape. The user will intuitibely be able to create various oval
spheres or complex box shapes by combining two circles or two
rectangles, respectively. Another method is to indicate a surface
of desired shapes by drawing spirals. In the future, we intend toincorporate these shape creation methods in our system. In
Addition to the shape creation, we intend to exploit the force
feedback generated by the kinesthetic display device in order toenable other manipulations of 3D shapes, such as transformingand painting.

[8]

T. Igarashi, S. Matsuoka, and M. Tanaka, “Teddy: A Sketching
Interface for 3D Freeform Design,” In ACM SIGGRAPH’99, pp.
409-416, 1999.
A. Rivers, T. Igarashi, and F. Durand, “2.5D Cartoon Models,” In
ACM Transactions on Graphics (SIGGRAPH 2010), 2010.
SensAble Technologies, Inc., FreeForm Systems,
http://www.sensable.com/products-freeform-systems.htm
T. Massie and K. Salisbury, “The PHANToM Haptic Interface: A
Device for Probing Virtual Objects,” In Proceedings of the ASME
Winter Annual Meetings, Symposium on Haptic Interfaces for
Virtual Environment and Teleoperation Systems, 1994.
A. D. Gregory, S. A. Ehman, and M. C. Lin, “inTouch: Interactive
Multiresolution Modeling and 3D Painting with a Haptic Interface,”
In Proceedings of the IEEE Virtual Reality 2000 Conference, pp. 4549, 2000.
M. Foskey, M. A. Otaduy, and M. C. Lin, “ArtNova: touch-enabled
3D model design,” ACM SIGGRAPH 2005 Courses, 2005.
M. Fiorentina, G. Monno, and A. Uva, “The Senstylus: a novel
rumble-feedback pen device for CAD application in virtual reality,”
In Proceedings of the 13th International Conference in Central
Europe on Computer Graphics, Visualization and Computer
Vision ’05 (WSXG 2005), 2005.
S. Kamuro, K. Minamizawa, N. Kawakami, and S. Tachi.
“Ungrounded Kinesthetic Pen for Haptic Interaction with Virtual
Environments,” In Proceedings of IEEE International Symposium on
Robot and Human Interactive Communication (IEEE RO-MAN2009), pp. 436-441, 2009.