Twinkle: Interface for using Handheld Projectors
to Intaract with Physical Surfaces
Takumi Yoshida∗
The University of Tokyo

Hideaki Nii
The University of Tokyo

Naoki Kawakami
The University of Tokyo

Susumu Tachi
The University of Tokyo
CameraProjector
Figure 1: (left):Concept of Twinkle, (center):Projected Image, (right):Prototype Device

Keywords: handheld projector, augmented reality

2 System Overview

1 Introduction

The proposed system comprises a handheld projector and a video
camera fixed to the projector. The system is quite simple and does
not need other motion-tracking systems. The camera captures aphysical surface illuminated by the projector. The camera has thefollowing two roles: to estimate the user’s motion and to recognize
the features of the physical surface. The process of estimationi ofthe user’s motion is described below. By recognizing the shape of
the area on which the light is projected, we can calculate the projector’s position relative to the surface and the distance between theprojector and the surface. Moreover, an optical flow technique enables the estimation of the direction and velocity of the motion of
the user’s hand on a two-dimensional surface. In order to recognize the features of the physical surface, various existing imageprocessing methods can be used. For example, by using a labelingprocess and pattern recognition, we can estimate where the object
is located on the surface and estimate its shape. These processes areexecuted in parallel and in real time. Then, images are generated
according to the user’s motion and the features of the surface, andthese images are projected onto the surface by the projector.

Recently, many small pocket-size projectors have been developed.
It is expected that in the near future, such projectors will be installedin portable devices. Meanwhile, intuitive interfaces that operate according to the user’s motion have been popular. Therefore, the interfaces that can be used for accessing information using handheld
projectors have been increasingly studied [Forlines et al. 2005][Caoet al. 2007]. However, these interfaces suffer from a number ofproblems. Some systems need motion-tracking systems in order tomeasure the position of the projector. Further, the surface whereimage is projected are limited to plain screen like a white wall.
The purpose of our study is to propose a novel interface for interaction with an arbitrary physical plane surface; here, we defineinteraction as the utilization of a physical surface to perform certain tasks. We call the interface “Twinkle”. We define a physicalsurface as a surface that exists in a physical environment and is notplain. Examples of such surfaces are a poster on a wall, figures orcharacters on a whiteboard, and a desk on which objects are placed.
When a user shines light from a handheld projector such as a flashlight onto a physical surface, pictures are displayed and sounds are
emitted according to the objects that are present on the surface andthe user’s motion. Figure 1 (left) shows the concept behind Twinkle.
Our method enables various applications. A few examples of theapplications of Twinkle are mentioned below. First, we propose
an interface for music composition and musical performance. The
pitch of a sound is determined by the size of the object illuminatedby the projector. The color of the object and the user’s motion determine the tone and the volume, respectively, of the sound. The
user can create melody and rhythm by laying out objects on a surface. This interface enables users to compose and play music on
the basis of intuition, i.e., they can compose and play music even if
they do not have knowledge of musical score. Next, we propose anAR annotation system. The system recognizes figures or characterson a surface, and information is presented near theose objects. Additionally, the proposed interface can be used in shooting games oraction games. In such games, real objects on a surface are regarded
as obstacles.
∗ e-mail:takumi yoshida@ipc.i.u-tokyo.ac.jp

We have developed a prototype device and implemented several applications. Figure 1 (center) shows a projected image when a user
plays music on the interface, and (right) shows a prototype device.

3 Conclusion
We proposed a novel interface for interaction with an arbitrary
physical plane surface, where the interaction involves the use ofhandheld projector. Further, we have developed a prototype device
and implemented several applications.

References
C AO , X., F ORLINES , C., AND BALAKRISHNAN , R. 2007. Multiuser interaction using handheld projectors. In Proceedings of the20th Annual ACM Symposium on User interface Software andTechnology, 43–52.
F ORLINES , C., BALAKRISHNAN , R., B EARDSLEY, P., V. BAAR ,J., AND R ASKAR , R. 2005. Zoom-and-pick: facilitating visualzooming and precision pointing with interactive handheld projectors. In Proceedings of the 18th annual ACM symposium onUser interface software and technology, 73–82.