An Initialization Tool for Installing
Visual Markers in Wearable Augmented Reality
Yusuke Nakazato, Masayuki Kanbara, and Naokazu Yokoya
Nara Institute of Science and Technology,
8916–5 Takayama, Ikoma, Nara, 630–0192, Japan

Abstract. It is necessary to precisely measure pose (position and orientation) of a user in order to realize an augmented reality (AR) systemwith a wearable computer. One of major methods for measuring user’spose in AR is visual marker-based approach which calculates them byrecognizing markers pasted up on the ceilings or walls. The method needs
3D pose information of visual markers in advance. However, much costis necessary to calibrate visual markers pasted up on the ceiling in awide environment. In this paper, an initialization tool for installing visual markers in wearable AR is proposed. The administrator is assisted
in installing visual markers in a wide environment by the proposed tool.
The tool calibrates alignment of visual markers which exist in the realenvironment with high accuracy by recognizing them in the images captured by a high-resolution still camera. Additionally, the tool assists theadministrator in repairing the incorrect pattern of marker using a wearable AR system.

1

Introduction

Recently augmented reality (AR) system using a wearable computer has received a great deal of attention as a method for displaying location-based information in the real world[1]. To realize a wearable AR system, the exact pose
(position and orientation) of the user is required. Especially in indoor environments, since a GPS cannot be used, various localization methods have been
investigated[2,3,4,5]. User’s pose is sometimes estimated by recognizing visual
markers pasted up on the ceilings or walls whose poses are known[6,7,8]. We
have proposed a localization method with an infrared camera and invisible visual markers consisting of translucent retro-reﬂectors as shown in Fig.1[9]. The
method can realize indoor localization without impairing the scenery in a wide
environment. In the visual marker-based approach such as in [9], the alignment
information of visual markers, which is 3D poses of markers located in real environment, is needed in advance.
In this paper, we focus on initialization methods of visual markers for wearable AR system. The initialization of visual markers means calibration of theiralignment in advance. The simplest method is calibration by hand using a ruler
and others, although the accuracy of calibration results is low. Another method
Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 228–238, 2006.
c Springer-Verlag Berlin Heidelberg 2006


An Initialization Tool for Installing Visual Markers(a) Invisible visual markers

229(b) Invisible visual markers with a ﬂash

Fig. 1. Positioning infrastructures with invisible visual markers placed on the ceiling.
Though it is diﬃcult to perceive the markers, these can be captured clearly with aﬂash.

using a laser rangeﬁnder can calibrate the poses of visual markers with high accuracy. However the laser rangeﬁnder is an expensive instrumentation in general.
Furthermore, these methods need much labor to calibrate the poses of visual
markers in a wide environment. On the other hand, the calibration methods using images of visual markers have been proposed[10,11,12]. Baratoﬀ, et al haveproposed a semiautomatic image-based calibration method requiring only minimal interaction[10]. Maeda, et al have calibrated all visual markers by capturingmarkers which include one reference marker automatically[11]. These methods
can work online, however the calibration results of markers’ poses do not havehigh accuracy, because the pose of marker is usually estimated based on propagation with some low resolution images. When a large number of visual markers
are installed as positioning infrastructures for user’s localization, the calibrationof visual markers with high accuracy is generally required. In this case, the oﬄine
calibration tool for wearable AR system is useful.
We focus on high accuracy calibration of visual markers’ poses used as positioning infrastructures. This paper proposes an initialization tool which assists on
administrator in installing visual markers. First, the administrator sets up visual
markers on the ceilings or walls and then captures images of visual markers bya high-resolution still camera. Secondly, the proposed tool generates alignment
information of visual markers by recognizing visual markers on the images captured by the administrator. Finally, the tool assists the administrator in repairing
duplicate patterns of visual markers using a wearable AR system. Because such
patterns of visual markers placed in the neighborhood sometimes cause failure ofuser localization. However, it is diﬃcult to conﬁrm such incorrect pattern from alarge number of visual markers by human beings. In particular, when the invisible visual markers as shown in Fig.1 are used, the task is more diﬃcult because
the markers are invisible. Therefore, the tool provides the administrator withimages containing indication of repairing markers using a wearable AR system.
This paper is structured as follows. Section 2 describes an initialization toolfor visual marker-based localization. In Section 3, experiments of infrastructure

230

Y. Nakazato, M. Kanbara, and N. Yokoya

Administrator

Initialization tool for installing visual markers

Installing visual markers
Capturing images
images, marker features, intrinsic camera parameters

Operating
1st phase of the tool

1st phase: Calibration of visual markers

Alignment information of visual markers
Repairing visual markers
using 2nd phase of the tool

2nd phase: Assistance in repairing theincorrect pattern of visual marker using AR

Updated alignment information of visual markers

Fig. 2. Flow diagram of initialization tool of visual markers

initialization with the proposed tool are described. Finally, Section 4 gives conclusion and future work.

2

Initialization Tool for Installing Visual Markers

In this section, the proposed initialization tool is explained in detail. The overview
of the proposed tool is illustrated in Fig.2. First, visual markers are installed as
positioning infrastructures and then high-resolution images of visual markers arecaptured by the administrator. Secondly, the tool calibrates poses of visual markers by using captured images and outputs calibration results as alignment information. Finally, using the calibration results, the tool provides the administrator
with augmented images in order to repair incorrect patterns of visual markers.
This tool consists of two phases: the ﬁrst phase is concerned with calibrationof visual markers and the second phase is to assist in repairing incorrect markers using a wearable AR system. In the ﬁrst phase, poses of visual markers arecalibrated from high-resolution images of markers. The administrator inputs (1)intrinsic camera parameters (focal length, pixel size, center of image, radial distortion factor coeﬃcient), (2) features of visual marker (frame length and width,
pattern size), and (3) images of visual markers. The tool estimates poses of visual
markers, and outputs this information as alignment information of visual markers. In the second phase, the tool assists the administrator in repairing incorrect
patterns of markers using alignment information of visual markers generated in
the ﬁrst phase. The tool provides the administrator with overlaid images containing indication of repairing visual marker and then outputs updated alignmentAn Initialization Tool for Installing Visual Markers

231

information of visual markers. Sections 2.1 and 2.2 explain about the ﬁrst andsecond phases, respectively.
2.1

First Phase: Calibration of Visual Markers

In this section, the ﬁrst phase of initialization tool is described.
Overview of First Phase of Initialization Tool. In the ﬁrst phase, thealignment information of visual markers is generated. This phase uses three forms
as shown in Figs. 3,4, and 5: an input, a conﬁrmation, and an output forms. Fig.3
illustrates the input form. The administrator inputs intrinsic camera parameters,
visual marker features, and images of visual markers. Then the administrator(1)
Input camera
parameters file(2)
Input marker
features file(3)
Input photos(4)
Start marker
recognition

Fig. 3. Input form in the ﬁrst phase(1)
List of
Recognizedmarkers(3)
Photo ofselected marker(2)
List of selectedmarker’s photos(5)
Assume allMarkers are ona single plane(4)
Modify the IDof the marker(6)
Start markers
calibration

Fig. 4. Conﬁrmation form in the ﬁrst phase

232

Y. Nakazato, M. Kanbara, and N. Yokoya(1)
Image ofmarkers’
localization(2)
Save alignment
information ofvisual markers

Fig. 5. Output form in the ﬁrst phase

Images，marker features，intrinsic camera parameters
Markers recognition
3D position estimationof markers
Estimation of cameras’ position and orientation
Estimation of markers’ position and orientation
Global optimization

Alignment information of visual markers

Fig. 6. Flow diagram of visual marker calibration

Fig. 7. Instancesof markers

dictates to start recognition of visual markers, and switches to the conﬁrmation
form. In the conﬁrmation form shown in Fig.4, the administrator conﬁrms resultof marker recognition. If misrecognized or duplicate visual markers exist, theadministrator modiﬁes this ﬁeld(Fig.4. (4)). Furthermore, if it is supposed thatall visual markers are on a single plane, the administrator sets a ﬂag on thisﬁeld(Fig.4. (5)). Then the administrator dictates to start calibration of visual
markers, and switches to the output form. Fig.5 illustrates the output form. The
form shows an image on which the calibrated visual markers are drawn and then
the tool outputs the alignment information of visual markers. This information
is used in the second phase.
Visual Markers Calibration. Fig.6 shows the ﬂow diagram of the process ofvisual markers calibration in the ﬁrst phase. First, visual markers in each image
in the input form are recognized. Secondly, one recognized visual marker is set
the 3D pose as a reference marker. Poses of cameras capturing the referenceAn Initialization Tool for Installing Visual Markers

233

visual marker are estimated. Then, position and orientation of visual markers
are estimated by estimated poses of the cameras. Finally, the poses of the visual
markers are optimized all over the images. The details of these processes aredescribed below.
Marker Recognition. We employ invisible visual markers[9] which consist of grid
points showing their IDs as shown in Fig.7. In the process of marker recognition,edges are detected from the input image using canny operator [13], then contoursof marker are retrieved. The area of original image corresponding to the contour
is binarized using a threshold which is determined by the discriminant analysis
method[14]. Four vertices of a square as a visual marker are detected from thearea, and its ID is recognized. The four vertices and centroids of grid points inthe marker are used as feature points in the process of 3D pose estimation ofmarkers described below.
3D Pose Estimation of Markers. First, one recognized visual marker is given its 3D
pose as a reference marker; vote that the reference marker determines the origin of
the world coordinate system. Next, using a Sato’s method[15], pose of camera capturing the reference marker are estimated. Then, poses of other markers captured
by camera known 3D position and orientation are estimated. Using the estimatedposes of markers, unestimated poses of cameras are estimated. These processesof pose estimation of camera and visual markers are iterated until all the markers’
poses are determined. In the iteration process, visual markers captured by multiple
images, which has a large reprojection error, are considered as those with duplicate patterns. The reprojection error is deﬁned as the sum of squared distances
between detected positions of marker’s feature points in the image and projected
positions of marker’s feature points in the world coordinate system. The poses of
duplicate visual markers are estimated individually. These visual markers with duplicate patterns are candidates for repairing in the second phase. Finally, 3D posesof cameras and visual markers are globally optimized so as to minimize the sum of
reprojection errors all over the images.
2.2

Second Phase: Assistance in Repairing Incorrect Markers UsingAR

In the second phase, our tool assists the administrator in repairing incorrect
patterns of markers using a wearable AR system. The administrator equips a
wearable computer, an HMD and an IR camera with IR-LEDs in order to capture the invisible visual markers as shown in Fig.8. The tool uses alignment
information of visual markers generated in the ﬁrst phase, and provides theadministrator with images containing indication of repairing markers as shownin Fig.8. The administrator’s view images are overlaid with IDs and contoursof recognized markers. If incorrect visual markers such as duplicate or lacked
patterns of markers exist in the administrator’s view, the tool provides the administrator images with the repaired pattern of marker. In case of Fig.8, theadministrator has modiﬁed ID No. 289 to No. 32. When the pattern of marker isrepaired by the administrator, the tool updates alignment information of visual
markers.

234

Y. Nakazato, M. Kanbara, and N. Yokoya

Alignment information of visual markers
HMD (SV-6[MicroOptical])IR Camera with
IR-LEDs[Original]

Wearable Computer(MP-XP7310[JVC])
Overlaid images with indication of repairing visual marker
Updated Alignment information of visual markers

Fig. 8. Overview of the second phase3

Experiments

First, we evaluated the accuracy of estimated position of visual markers in theﬁrst phase. In this experiment, invisible visual markers were set up on the ceiling. The size of the marker was 16cm. These markers are arranged at intervalsof 60cm on the ceiling plane. The 20 images captured by a digital still camera”OLYMPUS C-5050ZOOM” were used as shown in Fig.9. The size of these images was 2560 × 1920 pixels. The intrinsic camera parameters were estimatedby Tsai’s method [16] in advance. Since the visual markers were set up along
pattern of ceilings, the grand truths of the visual marker poses were calibrated
manually. When it is assumed that all markers are not located on the single
plane (no co-planar constraint), the projected points to 2D plane of estimatedpositions of visual markers’ four vertices are shown in Fig.10(a). Fig.10(b) shows

Fig. 9. Examples of input images (2560×1920 pixels)

An Initialization Tool for Installing Visual Markers

p1
p2

p1
p2
p3

3000

3000

p3

p4

2600

p4

2600

p1'
p2'
p3'

2200

p1'
p2'
p3'

2200

p4'

p4'
1800

1400

1400
y[mm]

1800

y[mm]

235

1000

1000

600

600

200
-200
-200

200
200

600

1000

1400

1800

x[mm]

(a) No-coplanar constraint.

2200

-200
-200

200

600

1000

1400

1800

2200

x[mm]

(b) Co-planar constraint.

Fig. 10. Results of marker calibration (pn:true position, pn’:estimated position)
Table 1. Average and standard deviation of marker calibration errors

average of absolute error[mm]standard deviation of error[mm]no co-planar constraint
x
y
z
9.9
11.6
16.9
7.7
9.7
13.6

co-planar constraint
x
y
z
6.0
3.7
0.0
5.2
4.6
0.0

estimated markers’ positions when it is assumed that all markers are located on
the single plane (co-planar constraint). In these ﬁgures, p1, p2, p3 and p4 show
true positions of visual marker’s four vertices, and p1’, p2’, p3’ and p4’ showcorresponding estimated positions. The average of absolute errors of markers’
positions and standard deviation of errors are shown in Table 1. In this experiment, we used a note PC (Pentium M 1.6GHz, Memory 512MB). When theno co-planar constraint is assumed, it took about 173 seconds for calibration of
the markers, and reprojection error was about 1.5 pixels. When the co-planar
constraint is assumed, the processing time of marker calibration was about 75
seconds, and reprojection error was about 1.7 pixels. From Table 1, we conﬁrmedthat the average error of estimated position was about 6mm and 17mm in caseof co-planar constraint and no co-planar constraint, respectively. In user localization, when we use a camera whose angle is 90 degrees and resolution is 640
× 480 pixels and then the user captures the visual markers on the ceiling 1.2m
away, the marker’s position error of 6mm corresponds to the error of about 1.6
pixels on the image. We can conclude that the accuracy obtained is suﬃcientfor user’s localization. Most visual markers were captured in about 3 images;
however some markers were captured only one time. These markers caused someerrors.

236

Y. Nakazato, M. Kanbara, and N. Yokoya

(a)

(b)

(c)

(d)

Fig. 11. Augmented images. These images are overlaid with IDs and repaired pattern ofmarker. The administrator modiﬁed the duplicate pattern of No.63 to No.47 according
to the indications given in these images.

In the experiment of the second phase, we conﬁrmed that proposed tool wasable to assist in repairing the patterns of markers. Fig.11 shows augmented
images with which the administrator is provided repairing the marker. From theresult of the ﬁrst phase, it was understood that No.63 is duplicated. Thus theadministrator modiﬁed the pattern to No.47 according to the indications given
in augmented images.

4

Conclusions

This paper has proposed an initialization tool for installing visual markers inwearable AR. In order to construct an environment for visual marker-based localization, the tool calibrates pose of visual markers by using images of markers
captured by high-resolution still camera. Furthermore, the tool assists the administrator in repairing incorrect patterns of markers by presenting augmented
images. In experiments, we have conﬁrmed that the proposed tool is able to estimate poses of markers with the high accuracy of about 6 mm when the co-planar

An Initialization Tool for Installing Visual Markers

237

constraint is assumed. We have also conﬁrmed that the tool is able to assist in
repairing incorrect patterns of markers using a wearable AR system. In future
work, we should carry out experiments of infrastructure initialization of visual
markers in a wide area. In order to calibrate poses of visual markers which havesome errors with more high accuracy, we will indicate the additional positions
at which the images should be taken using wearable AR system.

Acknowledgments
This research was supported in part by Strategic Information and Communications R&D Promotion Programme (SCOPE) of Ministry of Internal Aﬀairs andCommunications of Japan.

References
1. Höller, T., Feiner, S., Terauchi, T., Rashid, G., Hallaway, D.: Exploring mars: Developing indoor and outdoor user interfaces to a mobile augmented reality system.
Computers and Graphics 23(6) (1999) 779–785
2. Kourogi, M., Kurata, T.: Personal positioning based on walking locomotion analysis with self-contained sensors and wearable camera. Proc. 2nd IEEE/ACM Int.
Symp. on Mixed and Augmented Reality (ISMAR 03) (2003) 103–112
3. Tenmoku, R., Kanbara, M., Yokoya, N.: A wearable augmented reality systemusing positioning infrastructures and a pedometer. Proc. 7th IEEE Int. Symp. on
Wearable Computers (ISWC’03) (2003) 110–117
4. Vacchetti, L., Lepetit, V., Fua, P.: Combining edge and texture information forreal-time accurate 3D camera tracking. Proc. 3rd IEEE/ACM Int. Symp. on Mixed
and Augmented Reality (ISMAR 04) (2004) 48–57
5. Oe, M., Sato, T., Yokoya, N.: Estimating camera position and posture by using
feature landmark database. Proc. 14th Scandinavian Conf. on Image Analysis(SCIA2005) (2005) 171–181
6. Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P., Piekarski, W.:
First person indoor/outdoor augmented reality application: Arquake. Personaland Ubiquitous Computing 6(1) (2002) 75–86
7. Kalkusch, M., Lidy, T., Lnapp, M., Reitmayr, G., Kaufmann, H., Schmalstieg, D.:
Structured visual markers for indoor pathﬁnding. Proc. 1st IEEE Int. Augmented
Reality Toolkit Workshop (ART’02) (2002)
8. Naimark, L., Foxlin, E.: Circular data matrix ﬁducial system and robust image
processing for a wearable vision-inertial self-tracker. Proc. 1st IEEE/ACM Int.
Symp. on Mixed and Augmented Reality (ISMAR2002) (2002) 27–36
9. Nakazato, Y., Kanbara, M., Yokoya, N.: A localization system using invisible retroreﬂective markers. Proc. IAPR Conf. on Machine Vision Applications (MVA2005)(2005) 140–143
10. Baratoﬀ, G., Neubeck, A., Regenbrecht, H.: Interactive multi-marker calibration
for augmented reality applications. Proc. 1st IEEE/ACM Int. Symp. on Mixed
and Augmented Reality (ISMAR2002) (2002) 107–116
11. Maeda, M., Habara, T., Machida, T., Ogawa, T., Kiyokawa, K., Takemura, H.: Indoor localization methods for wearable mixed reality. Proc. 2nd CREST Workshop
on Advanced Computing and Communicating Techniques for Wearable Information Playing (2003) 62–65

238

Y. Nakazato, M. Kanbara, and N. Yokoya

12. Zauner, J., M. Haller, A.: Authoring of mixed reality applications including multimarker calibration for mobile devices. Proc. 10th Eurographics Symp. Virtual
Environments (EGVE2004) (2004) 87–90
13. Canny, J.: A computational approach to edge detection. IEEE Trans. PatternAnalysis and Machine Intelligence 8(6) (1986) 679–698
14. Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans.
Systems, Man, and Cybernetics SMC-9(1) (1979) 63–66
15. Sato, T., Kanbara, M., Yokoya, N.: 3-D modeling of an outdoor scene from multiple
image sequences by estimating camera motion parameters. Proc. 13th Scandinavian Conf. on Image Analysis(SCIA2003) (2003) 717–724
16. Tsai, R.Y.: An eﬃcient and accurate camera calibration technique for 3D machine
vision. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (1986)
364–374