Augmenting Real-world Objects
by Detecting “Invisible” Visual Markers
Yuki NotaYasuyuki Kono
Department of Informatics, School of Science and Technology
Kwansei Gakuin University2-1 Gakuen, Sanda 669-1337, Japan{bln28491, kono}@kwansei.ac.jp

ABSTRACTThis paper describes a vision-based method for detecting andidentifying real-world objects in front of the users. The proposed method employs “invisible” visual markers, which areinvisible from the users but visible from the ObjectCam2, ourcamera device. The ObjectCam2 is equipped with IR LEDs
around the lens that blink synchronously with the image capture, and the invisible markers consist of lucent and retroreflective paint. UbiComp applications can be implemented
without suffering from the power supply problem nor intrusive visual markers.
H5.1 [Information interfaces and presentation]: Multimedia Information Systems. - Artificial,augmented, and virtual realities.
ACM Classification:

General terms:
Keywords:

Design, Human Factors.

Real-world Computing, Retro-reflective Marker.

INTRODUCTION

This paper proposes a new method for identifying objects in
the real-world by detecting “invisible” visual markers by theObjectCam2 [8], a kind of active IR camera. It is required toidentify real-world objects to realize most UbiComp applications and also to detect their posture to realize augmented
reality (AR) applications. By identifying target objects infront of the user, for instance, such systems can augment
the objects with virtual annotations when they simply takea look at or come nearby the objects, i.e., they can be provided with the information associated with the objects. Typical object identification techniques in these fields employ either of radio tags[3], active optical tags[4], visual markers[2],or vision-based object recognition[6, 7]. Although tag-based
approaches are more reliable for identifying objects thanother approaches, the active tags which require power supply cannot be embedded in every object.
Although recognition/identification results from vision-based

Copyright is held by the author/owner(s).
UIST’08, October 19–22, 2008, Monterey, CA, USA
ACM 978-1-59593-975-3/08/10.

methods tend to be more ambiguous than those of tag-based
approaches, vision-based methods have advantage in simplicity of their device configuration. Especially use of visual markers increases robustness and reduces computation
time. Typical marker-based methods like ARToolkit[2] employ square black and white images with a simple patterninside as a marker and a unique pattern represents the IDof the object/region where the marker is placed. One of thebiggest problems for introducing visual markers into daily
life is their intrusiveness.
Nakazato et al. proposed a localization method for wearable AR systems that employs invisible markers consistingof translucent retro-reflectors[5]. In addition to the ordinary
color camera for scene capturing it employs an IR cameraand blinking IR LEDs for localization. Invisible markers are
set on the ceilings or walls and the IR camera captures the reflection of the IR LEDs. Hanhoon et al. proposed a method
for AR applications that detects invisible markers drawn withIR Pen[1]. Their camera system is similar to our older version of ObjectCam [7], i.e., a color camera for scene capturing and an IR camera for tracking are positioned in eachside of a cold beam splitter. Although these approaches arefree from visual intrusiveness, both employ two cameras that
increase size, weight, and computational requirements. Although translucent retro-reflectors can be placed to materials with less texture such as ceilings used in [5], translucent
markers on textured objects are also intrusive.
Our proposed method employs the single color camera, called
ObjectCam2, which has IR LEDs blinking synchronouslywith the image capture and can capture both the real scene
and the reflection of the IR illumination for identifying theinvisible markers consist of lucent and retro-reflective paint.
Our method has less substance limitation than the method using IR Pen, because each marker is implemented as a directly
sprayed pattern on to the surface of an object.
OBJECT IDENTIFICATION AND APPLICATION
ObjectCam2

The ObjectCam2 was originally developed as a wearable
camera for detecting handheld objects [8]. It is a kind ofactive IR camera with an IR LED array in its front and isequipped with the CMOS image sensor whose frame rate is90fps. The image sensor has no IR cutoff filter so that theObjectCam2 can capture an image that has luminance withinboth visible and IR light range. It controls blinking in the IRbackground. Figure 2(a) shows the IR reflection image ofthe scene. The system detects invisible markers and identifies them from image (a) by the following process:1. binarizes the reflection image and eliminates noise by recursively applying erosion and dilation transform,(a) LEDs On(b) LEDs Off

Figure 1: Real-world objects annotated by invisible
markers

2. detects marker candidate region(s) by finding closing contours and approximating each of them using Douglas-Peucker
polyline simplification algorithm, and selects the largest
candidate region,
3. normalizes the size of the region image (Figure 2(c)), and4. matches the region image with marker templates and identifies the marker.
When a marker is detected and is successfully identified, thesystem passes its ID to the application.
Application

Our prototype application shows the web page associatedwith the identified real-world object as described before. In
Figure 2, for instance, the object is successfully identified asthe booklet of Munch exhibition held in a Japanese museum
so that the application shows the exhibition web page.
ACKNOWLEDGMENTS
Figure 2: System Diagram

The authors thank Shinichi Yoshimura and Shigeyuki Baba
at Sony Corp. for implementing ObjectCam2.

LED array synchronously to the image capture, and captures
both two color images with reflected active IR light (Figure1(a)) and a color image without the lights (Figure 1(b)) in1/30 sec. By subtracting image (b) from image (a) on theFPGA circuit implemented in the ObjectCam2, it can generate both the IR reflection image and the color image of thescene , i.e., image (b), at 30fps.

REFERENCES

Invisible Marker

3. Lamming, M. and Flynn, M. Forget-me-not: Intimate Computing in Support of Human Memory. Proc.
FRIEND21: International Symposium on Next Generation Human Interface, pp.125-128, 1994.

The ObjectCam2 captures the reflection image of the IR illuminant at 30fps as described before. Because the retroreflector reflects a light toward a light source, its reflection ofIR LEDs which are set around the lens are clearly capturedby the ObjectCam2. Our invisible markers consist of lucent
and retro-reflective paint which contains retro-reflective glass
beads in oil-based solvent. Because a marker pattern is directly sprayed onto the surface of a target object, it can beapplied to almost any substances. Figure 1 shows real-world
objects of which invisible markers are set on the surface. Figure 1(a) is the image captured when the lights are on and (b)is off. The materials are a booklet made of coated paper (upper left), a wallpaper (upper right), a nylon top mouse pad(lower left), and a cotton luncheon mat (lower right). The
marker patterns can be extracted by subtracting image (b)from image (a).
Marker Detection and Identification

Figure 2 shows the diagram of our prototype application thatdetects target objects in a scene and shows the web page associated with the object whose marker occupies the largest
region in the scene. Figure 2(b) shows the scene which contains the same booklet as upper left of Figure 1 with complex

1. Hanhoon, P. and Jong-Il, P. Invisible marker based augmented reality system. SPIE Visual Communications andImage Processing, Vol. 5960, pp. 501-508, 2005.
2. Kato, H. and Billinghurst, M. Marker tracking and HMD
calibration for a video-based augmented reality conferencing system. Proc. IWAR99, Oct. 1999.

4. Matsushita, N., et al. ID CAM: A smart camera for scene
capturing and ID recognition. Proc.ISMAR2003, pp.227234, 2003.
5. Nakazato, Y., Kanbara, M., and Yokoya, N. Localizationof wearable users using invisible retro-reflective markers
and an IR camera. Proc. SPIE Electronic Imaging, Vol.
5664, pp. 563-570, Jan. 2005.
6. Schiele, B., Jebara, T., and Oliver, N. Sensory Augmented Computing: Wearing the Museum’s Guide.
IEEE Micro Journal, 2001.
7. Ueoka, T., Kawamura, T., Kono, Y., and Kidode, M. I’m
Here! : A wearable object remembrance support system.
Proc. MobileHCI 2003, pp.422-427, 2003.
8. Ueoka, T., et al. Wearable camera device for supporting object-triggered memory augmentation. Proc. 3rd
CREST/ISWC Workshop on Wearable Information Playing, pp. 46-53, 2004.