Localization of Wearable Users
Using Invisible Retro-reflective Markers and an IR Camera
Yusuke Nakazato, Masayuki Kanbara and Naokazu Yokoya
Graduate School of Information Science, Nara Institute of Science and Technology,
8916–5 Takayama, Ikoma, Nara, 630–0192, Japan
ABSTRACTThis paper describes a localization method for wearable computer users. To realize applications of wearable
computers like a navigation system, the position of a user is required for location-based services. Many localization methods in indoor environments have been proposed. One of the methods estimates user’s position usingIR beacons or visual markers. However, these methods have same problems concerning power supply and/or
undesirable visual eﬀects. In order to avoid the problems, we propose a new localization method which is basedon using an IR camera and invisible markers consisting of translucent retro-reﬂectors. In the proposed method,
to extract the regions of the markers from the captured images stably, the camera captures the reﬂection of IR
LEDs which are ﬂashed on and oﬀ synchronously.
Keywords: Retro-reﬂective Marker, IR Camera, Localization, Wearable Computer

1. INTRODUCTION
Since computers have made remarkable progress in resent years, a wearable computer can be realized1 . At thesame time, the augmented reality (AR) technique which merges the real and virtual worlds has received a great
deal of attention as a new method for displaying location-based information in the real world2–4 . Therefore,
AR systems using wearable computers like navigation systems5–8 are proposed. To realize an AR system using
wearable computer, the exact position and orientation of a user are required. Especially in indoor environments,
since a GPS can not be used, many localization methods have been proposed9, 10 . One of the methods estimates
the user’s position using infrared markers11, 12 . The methods specify the user’s position using the position IDs
received from IrDA markers which compose positioning infrastructures. Tenmoku, et al11 have estimated user’s
position using infrared markers and pedometer, and the orientation of the user by gyro sensor. Maeda, et al12developed a hybrid tracking system that estimates more exactly the position and orientation of the user by
combining a gyro sensor with a stereo camera which captured infrared markers. Another method estimated theuser’s position by recognizing the visual markers which are pasted up on the ceilings or walls13, 14 . Baratoﬀ,
et al13 used ARToolKit15 square markers, as shown Figure 1(a) . Naimark, et al14 developed a system that
estimates more stably the position and orientation of the user by combining an accelerometer with a camera
which captured circular markers as shown Figure 1(b) . However, these methods have some problems concerning
power supply of positioning infrastructure and/or undesirable visual eﬀects.
In order not to impair the scenery, we propose a new localization method using invisible markers that consistof translucent retro-reﬂectors. In the proposed method, translucent retro-reﬂective markers, which are invisible
and do not need power supply, are captured by an IR camera. In order to avoid the inﬂuence of infrared other than
the reﬂection from retro-reﬂective markers, infrared LEDs are ﬂashed on and oﬀ continuously. Thus, the images
containing the markers are captured synchronized with it. The regions of the markers are robustly extracted
from the diﬀerence between images as LED on and oﬀ. Herewith, the position of a user can be estimated without
power supply of infrastructures and undesirable visual eﬀects in the real scene.
This paper is structured as follows. Section 2 describes a localization system using invisible markers and IR
camera. In Section 3, experimental results with a prototype system are described. Finally, Section 4 describes
conclusion and future work.
Further author information: (Send correspondence to Y. N.)
Y. N.: E-mail: yuusu-n@is.naist.jp, Telephone: +81 743 72 5296
M. K.: E-mail: kanbara@is.naist.jp, Telephone: +81 743 72 5292
N. Y.: E-mail: yokoya@is.naist.jp, Telephone: +81 743 72 5290

(a) Augmented reality system using squaremarkers (Baratoﬀ, et al13 ).

(b) Localization system using camera and accelerometer (Naimark, et al14 ).

Figure 1. Localization methods using visual markers.

2. USER LOCALIZATION USING INVISIBLE MARKERS
In this section, our proposed localization system is described in detail. In Section 2.1, the invisible markers aredescribed. Section 2.2 discusses marker patterns, and ﬁnally Section 2.3 explains a method for estimation ofposition and orientation.

2.1. Invisible Markers
The markers are set up on the ceilings or walls in indoor environments as infrastructures. When visual markers
are set up on the ceilings, the markers impair the scenery as shown in Figure 2(a). Figure 2(b) shows the scene
where invisible markers consisting of translucent retro-reﬂectors are set up. Since the markers are translucent, it
is diﬃcult for a user to observe the markers. However, when the image is captured with a ﬂash, the markers can
be clearly observed as shown in Figure 2(c). Because the retro-reﬂector reﬂects a light toward a light source, its
reﬂection can be captured clearly by the camera which is located near the ﬂashing light.

(a) Visual markers.

(b) Invisible markers.

(c) Invisible markers with a ﬂash.

Figure 2. Markers of positioning infrastructures.

Figure 3. Instances of markers (N = 4) (the black part illustrates a retro-reﬂector).

2.2. Pattern of markers
Figure 3 illustrates instances of marker patterns. In this ﬁgure, the black part illustrates the retro-reﬂector, andthe marker has a square frame. To generate a marker pattern, the inside of the frame contains evenly spaced
N × N grid points, and small squares are allocated on the grids. To determine the direction of the marker
uniquely, one small square is always allocated to one of four corners of N × N grids, and other three corners are2blank. Therefore, the total number of IDs associated the patterns is 2N −4 .

2.3. Estimation of position and orientation
Figure 4 illustrates the overview of the proposed localization system. Invisible markers consisting of translucent
retro-reﬂectors are set up on the ceilings or walls. The user equips the head with an infrared camera shown inFigure 5 upward for capturing images. The camera captures the reﬂection of the infrared LEDs that are attached
to it. The reﬂection of retro-reﬂector can be captured clearly by the infrared camera.

Translucent Retro-reflective
MarkerReflection

Infrared Light

RS-232C

Wearable Computer(MP-XP7310 [Victor])

Video Capture Unit (NV-UT200 [NOVAC])
IR LEDs and IR camera

Figure 4. Overview of localization system.

Infrared Camera

IR-LED switch off

Infrared LED

Capturing image
IR-LED switch on
Capturing image
47mm

Image subtraction
Marker recognition &Position estimation

72mm

Figure 5. Infrared camera and infrared LEDs.

Figure 6. Flow diagram of capturing markers.

However, the camera also captures lights other than the infrared LEDs; for examples, ﬂuorescent and sunlight. To avoid such a problem, infrared LEDs are ﬂashed on and oﬀ synchronously under control of a wearable
computer as shown in Figure 6. In the image without infrared light from the infrared LEDs, invisible markers
are not captured. On the other hand, in the image with infrared light from the infrared LEDs, these are clearlycaptured. By calculating the diﬀerence between the images with and without infrared light from the LEDs,
the inﬂuence of infrared other than the reﬂection from retro-reﬂective markers can be eliminated. From thesubtraction images, the regions of the markers are extracted, and IDs associated with the markers are recognized.
To extract the regions of markers, ARToolKit15 can be used. In addition to identifying the received marker,
it is possible to estimate the relative position and orientation of camera with respect to the marker coordinate
system from the four vertices of a square marker of known size using a standard computer vision technique.

3. EXPERIMENTS
We have carried out experiments with the proposed localization system. View angle of the infrared camera
shown in Figure 5 is 92.6◦ , and six infrared LEDs are attached around the camera. We made a circuit which
controls the LEDs using RS-232C serial communication with a PC. The used PC is a mobile computer “InterLink
MP-XP7310 (Pentium M 1GHz)”, and the input image size is 320 × 240 pixels. The distance between the camera
and the ceilings is about 120cm. In Section 3.1, recognition results of invisible markers are described. Section
3.2 gives experimental results about accuracy of the localization.

3.1. Results of marker recognition
In this experiment, we conﬁrm that the proposed system recognizes position IDs from the markers without power
supply of infrastructures and undesirable visual eﬀects in the real scene . When the infrared LEDs are switch-oﬀ,
retro-reﬂective markers are not captured as shown in Figure 7(a). On the other hand, when infrared LEDs areswitch-on, the markers are clearly captured as shown in Figure 7(b). Figure 8 shows results of marker recognition.
These images were generated by recognizing subtraction images between Figures 7(a) and 7(b). Note that theposition IDs associated with the markers, and cones representing the direction of user are superimposed on thesubtraction images. We conﬁrm that the ID of marker can be recognized. The system can calculate the relative
position and direction of camera with respect to the marker coordinate system. The processing rate is about
15 frames per second, because two images when the infrared LEDs are switch-on and switch-oﬀ are required forrecognition.

(a) Captured images when infrared LEDs are switch-oﬀ.

(b) Captured images when infrared LEDs are switch-on.
Figure 7. Images obtained by infrared camera.

Figure 8. Results of marker recognition.

30[cm]-0[cm]

60[cm](0,0)

210[cm]270[cm]150[cm]240[cm]300[cm]120[cm]180[cm]y52[cm]
104[cm]x
Figure 9. Layout of markers.

3.2. Accuracy evaluation of localization results
In this experiment, we evaluate accuracy of the proposed system. The markers are set up on the ceilings, as
shown in Figure 9. We employ two diﬀerent size markers to recognize the markers even if the distance between acamera and markers is changed. The edge length of the small marker is 16cm and the size of its square element
allocated on the grids is 1cm × 1cm. On the other hand, the large marker’s edge length is 28cm, and the size ofits square element is 2cm × 2cm. The both markers’ frame width is 1cm. The number of grids N in the marker
pattern is 4 as in Figure 3.

MarkerCamera

Figure 10. Estimated camera position and orientation.

In this experiment, x-y plane represents the ceiling plane. And the z axis corresponds to the vertical direction.
The pitch ,roll and yaw are the rotation angles of x, y, and z axis in the marker coordinate system, respectively.
The distance between the camera and the ceiling is about 120cm, and the camera which looks toward theceiling, moves to (60,300,120) from (60,20,120) in parallel to the ceiling. Figure 10 illustrates estimated camera
position and orientation when the camera translates. Figures 11 and 12 show the error of estimated positionand orientation, respectively. We can conﬁrm from the experiment that the error of estimated camera positionand orientation are at most about 20cm and 10◦ , respectively. The average of estimated position and orientation
errors were about 6cm and 3.3◦ , respectively. We conclude that the accuracy of estimated camera position iscomparable with the case of using conventional visual markers.

xyz

Error of Position[cm]30201001265176101-10-20

FramesFigure 11. Error of estimated camera position.

Error of Orientation[°]yawrollpitch

15.0
10.0
5.0
0.0
-5.0

12651

76

-10.0
-15.0

FramesFigure 12. Error of estimated camera orientation.

101

4. CONCLUSION
This paper has proposed a new localization method using invisible markers and infrared camera. In our method,user’s position is identiﬁed by recognizing translucent retro-reﬂective markers which are illuminated from infrared
LEDs. In order to eliminate the inﬂuence of infrared other than the reﬂection from retro-reﬂective markers, theinfrared LEDs are ﬂashed on and oﬀ synchronously with image capturing.
In the experiments, we have conﬁrmed that the accuracy of estimated camera position and orientation arecomparable with the case of using conventional visual markers. In future work, we should carry out experiments
at the extensive indoor environments. We will also improve the accuracy by considering the reliability of themarker’s position on a captured image.

ACKNOWLEDGMENTS
This research was supported in part by Core Research for Evolutional Science and Technology (CREST) Program“Advanced Media Technology for Everyday Living” of Japan Science and Technology Agency (JST).

REFERENCES
1. S. Mann, “Wearable computing: A ﬁrst step toward personal imaging,” IEEE Computer 30(2), pp. 25–32,
2002.
2. R. Azuma, “A survey of augmented reality,” Presence Vol. 6(No. 4), pp. 355–385, 1997.
3. M. Kanbara, T. Okuma, H. Takemura, and N. Yokoya, “A stereoscopic video see-through augmented reality
system based on real-time vision-based registration,” Proc.IEEE Int. Conf. on Virtual Reality 2000 , pp. 255–262, 2000.
4. S. Julier, M. Lanzagorta, Y. Baillot, L. Rosenblum, S. Feiner, T. Höller, and S. Sestito, “Information ﬁltering
for mobile augmented reality,” Proc. 1st IEEE/ACM Int. Symp. on Augmented Reality , pp. 3–11, 2000.
5. H
6. R. Malaka and A. Zipf, “Deep map - challenging it research in the framework of a tourist informationsystem,” Proc. 7th Int. Congress on Tourism and Comm. , pp. 15–27, 2000.
7. P. Daehne and J. Karigiannis, “Archeoguide: System architectureof a mobile outdoor augmented reality
system,” Proc. 1st IEEE/ACM Int. Symp. on Mixed and Augmented Reality , pp. 263–264, 2002.
8. R. Tenmoku, Y. Nakazato, A. Anabuki, M. Kanbara, and N. Yokoya, “Nara palace site navigator: Deviceindependent human navigation using a networked shared database,” Proc.10th Int. Conf. on Virtual Systems
and Multimedia , pp. 1234–1242, 2004.
9. M. Kourogi and T. Kurata, “Personal positioning based on walking locomotion analysis with self-containedsensors and wearable camera,” Proc. 2nd IEEE/ACM Int. Symp. on Mixed and Augmented Reality , pp. 103–112, 2003.
10. L. Vacchetti, V. Lepetit, and P. Fua, “Combining edge and texture information for real-time accurate 3dcamera tracking,” Proc. 3rd IEEE/ACM Int. Symp. on Mixed and Augmented Reality , pp. 48–57, 2004.
11. R. Tenmoku, M. Kanbara, and N. Yokoya, “A wearable augmented reality system using positioning infrastructures and a pedometer,” Proc. IEEE Int. Symp. on Wearable Computers , pp. 110–117, 2003.
12. M. Maeda, T. Ogawa, K. Kiyokawa, and H. Takemura, “Tracking of user position and orientation by stereo
measurement of infrared markers and orientation sensing,” Proc. IEEE Int. Symp. on Wearable Computers, pp. 77–84, 2004.
13. G. Baratoﬀ, A. Neubeck, and H. Regenbrecht, “Interactive multi-marker calibration for augmented realityapplications,” Proc. 1st IEEE/ACM Int. Symp. on Mixed and Augmented Reality , pp. 107–116, 2002.
14. L. Naimark and E. Foxlin, “Circular data matrix ﬁducial system and robust image processing for a wearable
vision-inertial self-tracker,” Proc. 1st IEEE/ACM Int. Symp. on Mixed and Augmented Reality , pp. 27–36,
2002.
15. H. Kato and H. Billinghurst, “Marker tracking and hmd calibration for a video-based augmented reality
conferencing system,” Proc. 2nd IEEE/ACM Int. Workshop on Augmented Reality , pp. 85–94, 1999.