ARForce
Kensei Jo † ∗Yasuaki Kakehi ‡Kouta Minamizawa †Hideaki Nii †Naoki Kawakami †Susumu Tachi ††‡The University of Tokyo
Keio University / Presto, Japan Science and Technology Agency

Figure 1: Conceptual Image.

1Introduction

In recent years, the usage of augmented reality (AR) systems has
become common. AR achieves the combination of real-world andcomputer-generated data. ARTookKit[Kato and Billinghurst 1999]
is one of the prominent examples of an AR system; it measures the3D positions and IDs of physical markers and combines computergenerated graphics with real video images. Users can move or rotate the virtual graphics by relocating the physical markers; however, it is impossible to touch or manipulate the virtual graphics
directly using the users’ fingers.
The objective of this research is to manipulate virtual graphics using fingers, for example, touching, pushing, twisting, pinching, andexpanding the virtual graphics in midair. To achieve our objective,
we require an interface that has the following three features. First,
the interface must usable in midair. Second, it must recognize multiple simultaneous touch points. Third, it must measure both themagnitude and direction of force applied using fingers.
Therefore, the aim of our study to develop a wireless interface thatmeasures the distribution of force vectors applied by fingers, in order to construct AR environments in which users can manipulate
virtual graphics by the use of natural finger motions. To achieve
this aim, we propose a novel marker-based interface termed ”ARForce.” This interface can detect the distribution of force vectors on
its surface as well as its 3D position, rotation, and ID by using avision-based approach. ARForce enables users to observe overlaid
virtual images and control them using fingers. In the future, weintend to measure information precisely and control virtual objects
in the same manner as real objects. As a result, we will develop anovel computer-human interface that accepts a highly natural input.

2

Background and Related WorkIn the field of Computer Human Interaction, various types ofinteractive systems have been proposed that allow users to manipulate virtual images using fingers or hands of users. CyberGlove[Immersion ] and Soap[Baudisch et al. 2006] are representative researches of the finger pointing device that works in midair.
∗ e-mail:kensei jo@ipc.i.u-tokyo.ac.jp

Figure 2: Simplified diagramof GelForce.

Figure 3: ForceTile.

Rekimoto et al. have also proposed a hand-held pointing device thatuse tilt as the input method[Rekimoto 1996].
On the other hand, in a field of Augmented Reality, there are a lot ofresearches that combine the virtual data to real world. In this field,
many systems utilized 2D marker[qrc ] to measure IDs and positions[Kakehi et al. 2007; Woods et al. 2003]. By using a visionbased marker recognition method, we can detect the position, rotation and ID of object easily in real-time. In addition to these
input, some systems tried to recognize richer information about
users’ input. Active CyberCode[Ayatsuka and Rekimoto 2006] allows a user to give commands by putting his/her finger on a printed
button beside the code. EnhancedDesk[Koike et al. 2000] recognizes not only 2D markers but also gestures of the user. The Haptic
Hand[Kohli and Whitton 2005] uses special device in addition to2D marker, and allows user to touch virtual objects.
Toward these systems, we propose a novel tangible interface “ARForce that works as a 3D marker and pointing device. This interface allows user to manipulate the virtual images using not only 3D
position and ID but also the distribution of force vectors without
using electric devices. Furthermore, note that the markers on ourinterface are invisible for users unlike many of other marker-basedsystem.
In the implementation of ARForce, we adopt the method of our previous project “Gelforce[Kamiyama et al. 2005; Sato et al. 2007]
to measure a distribution of force vectors. Its sensor consists of atransparent elastic body and two layers of colored markers within

Elastic Body(transparent)
Elastic Body(black)

Camera and IR Emitter
15Case(IR filter)
Square MarkerForce Markers

100100[mm]

Input Device.
Figure 5: Structure of proposed interface.

the transparent and black elastic bodies are deformed and the force
markers that are placed in the transparent elastic body are moved. Inorder to capture the 3D position of the interface, a square marker isattached to the acrylic case. Each interface has different patterns offorce markers, which enables ID discrimination. The force markers
and square marker are composed of retroreflective materials.
The acrylic case acts as an IR filter that blocks the visible light
spectrum and transmits IR rays. Therefore, the interface appears
black, and it is impossible to observe the inside of the interface
with the naked eye.
Because many CCD cameras have optical filters thatblock IR rays, it is impossible to capture the force markers using
such cameras. Therefore, we remove the filter from our camera.
An IR emitter is placed near the camera because the force markersand the square marker are composed of retroreflective materials.
Fig. 6 shows the captured image of the interface.

CameraFigure 4: (Top) Appearance of proposed system. (Bottom) Captured image and force vectors. The green arrows show the force
vectors applied by the user.

the body (Fig. 2). When a force is applied to the body’s surface, we
optically measure the internal strain of the body through the movement of the markers. Finally, force vectors are calculated from thestrain by an elastic theory. Recently, we have also developed a tangible interface named ForceTile[Kakehi et al. 2008](Fig. 3) using
the method of GelForce. However, there is a clear difference thatForceTile is designed only for tabletop display, and ARForce canbe used in 3D space.

3System Overview

3.2 SoftwareThis system calculates the distribution of force vectors using thesame method as that of GelForce. GelForce captures the displacement of force markers, which is caused by the force applied by theuser. However, in our proposed system, the positions of the force
markers depend not only on the force vectors but also on the position of the device. Therefore, we use an algorithm comprising thefollowing steps.
1. Search for the square marker in the captured image and rec-Fig. 4 (top) shows the appearance of the ARForce system. A usercontrols the tile-shaped interface and applies force to the bottomof the interface with his/her fingers. A camera placed above theuser captures the image of the interface using infrared (IR) light.
Using a computer vision technique and the elastic theory, we canmeasure the 3D positions and IDs of interfaces and the distributionof force vectors in real time. Fig. 4 (bottom) shows the measured
force vectors and captured image. The green arrows show the force
vectors applied using fingers.

3.1

HardwareFig. 5 shows the structure of the proposed tileshaped interface. The outer body of the interface is enclosed inan acrylic case. The inside of the interface contains a transparent
elastic body. When the user applies force to the interface, both

Input Device

Figure 6: (left) Camera image without IR light. (right) Camera
image under IR light. The inside of the interface is invisible to thenaked eye. However, the use of the camera and IR light enables us
to capture the view inside the interface. Each interface has different
dot patterns, which enables ID detection.

ognize the ID from the pattern of the force markers.
2. Calculate the 3D position of the interface by measuring thecorner position of the square marker in the picture coordinatesystem.
3. Measure the relative displacement of the force markers, whichis caused by the force applied by the user’s finger．
4. Calculate the force vectors using the same method as that ofGelForce.

4 Apprications.
To demonstrate these functions mentioned above, we have already
implemented several prototype applications as followings.
Figure 8: Movie Player.
In this application, we use ARForce as avirtual object viewer to demonstrate the recognition of finger gestures. This application involves the overlay of virtual images on
the live image like ARToolKit. Furthermore, when user expands
or twists the input surface of the device, the virtual images are deformed according to the gestures. For example, when the user expands the surface strongly, the graphics enlarge at high speed. Fig.
7 shows snapshots of this application.

Virtual Objects Viewer

position and tilt of the spaceship corresponds to the interface’s position and posture. In addition, users can shoot beams by pushing theinput surface of the ARForce interface. The direction of the beamchanges according to the direction of force applied by the user.

Figure 9: Shooting Game.

4.1 Experience for Attendees.
In the site of SIGGRAPH Asia 2008, we will demonstrate the applications mentioned above. We are going to use some cameras andmultiple interfaces. Because ARForce discriminates IDs, multiple
attendees can join the interactions at one time. In addition, we plan
to install projectors or monitors to show the virtual images to theattendees.
Figure 7: Application of Virtual Objects Viewer.

Movie Player Fig. 8 shows snapshots of the movie player. In thisapplication, users can control the time parameter of the movies displayed on the interface with their twisting gestures. When the user
twists it in a clockwise direction, the movie plays in forward direction; and in a counterclockwise, the movie is rewound. In addition,
the playback speed changes according to the twisting speed.

We have also implemented an entertainment application for multiple users. Fig. 9 shows snapshots of the shooting
game. Users can operate virtual spaceship using the interface. TheShooting Game

5 Conclusion and Future Works
In this paper, we have proposed a novel marker-based interface thatcan measure not only the 3D positions and IDs of markers but also
the distribution of the magnitude and direction of force applied onto
it. Using this interface, users can manipulate virtual graphics which
are combined to the real-world scene with natural finger motions.
In the future, we have following plans: First, we will keep improving the method and algorithm to measure more precise force vectors. For example, we plan to utilize cameras with high resolution
or wide dynamic ranges. Secondly, we will implement a series of
new AR applications including games, simulations, media art, andscientific visualizations. Thirdly, we will extend the variety of theinterface so that they vary in shape, size, and material.

References
AYATSUKA , Y., AND R EKIMOTO , J. 2006. Active CyberCode:A Directly Controllable 2D Codes. In CHI 2006 Extended Abstract, pp490–495.
BAUDISCH , P., S INCLAIR , M., AND W ILSON , A. 2006. Soap:a Pointing Device that Works in Mid-Air. In UIST 2006 (Technote).
I MMERSION. CyberGlove. http://www.immersion.com/3d/products/cyber glove.php.
K AKEHI , Y., NAEMURA , T., AND M ATSUSHITA , M. 2007. Tablescape Plus: Interactive Small-sized Displays Upstanding on
Tabletop Display. In 2nd Annual IEEE International Workshop
on Horizontal Interactive Human-Computer Systems (Tabletop2007).
K AKEHI , Y., J O , K., S ATO , K., M INAMIZAWA , K., N II , H.,K AWAKAMI , N., NAEMURA , T., AND TACHI , S. 2008.
Forcetile: Tabletop tangible interface with vision-based force
distribution. In ACM SIGGRAPH 2008 New Tech Demo.
K AMIYAMA , K., V LACK , K., M IZOTA , T., K AJIMOTO , H.,K AWAKAMI , N., AND TACHI , S. 2005. Vision-based sensor for
real-time measuring of surface traction fields. IEEE Computer
Graphics and Applications 25, 1, 68 – 75.
K ATO , H., AND B ILLINGHURST, M. 1999. Marker tracking andHMD calibration for a video-based augmented reality conferencing system. In Proceedings 2nd IEEE and ACM International
Workshop on Augmented Reality (IWAR’99), pp85 – 94.
KOHLI , L., AND W HITTON , M. 2005. The haptic hand: providinguser interface feedback with the non-dominant hand in virtualenvironments. In Proceedings of Graphics Interface 2005.
KOIKE , H., S ATO , Y., KOBAYASHI , Y., T OBITA , H., AND
KOBAYASHI , M. 2000. Interactive Textbook and Interactive
Venn Diagram: Natural and Intuitive Interface on Augmented
Desk System. In Conference Extended Abstracts on Human Factors in Computer Systems (ACM CHI 2000), pp121–128.
Qrcode.
http://www.denso-wave.com/qrcode/index-e.html.
R EKIMOTO , J. 1996. Tilting Operations for Small Screen Interfaces. In User Interface and Software Technology (UIST’96).
S ATO , K., M INAMIZAWA , K., K AWAKAMI , N., AND TACHI , S.
2007. Haptic Telexistence. In 34th Int. Conf. On Computer
Graphics and Interactive Techniques (ACM SIGGRAPH 2007),
Emerging Technologies.
W OODS , E., M ASON , P., AND B ILLINGHURST, M. 2003. MagicMouse: an Inexpensive 6-Degree-of-Freedom Mouse. In Proceedings of Graphite 2003.