Geometric and Photometric Registration for Real-time Augmented Reality
Masayuki Kanbara and Naokazu Yokoya
Graduate School of Information Science, Nara Institute of Science and Technology
8916-5 Takayama-cho, Ikoma-shi, Nara 630-0101, JAPANfkanbara, yokoya g @is.aist-nara.ac.jp

Abstract
This paper proposes an augmented reality system withcorrect representation of shading and shadow. To realize aseamless augmented reality system, we need to resolve someproblems. Especially, the geometric and photometric registration problems are important. These problems require theposition of light sources and user’s viewpoint. The proposedsystem resolves the problems using a 3D marker which combines a 2D square marker and a mirror ball. The 2D marker
and the ball are used to estimate the relationship between
the real and virtual worlds and the positions of light sourcesin the real world, respectively.

1. Introduction
Augmented reality produces an environment in which
virtual objects are superimposed seamlessly on user’s viewof the real environment. Augmented reality has received agreat deal of attention as a new method for displaying information or increasing the reality of virtual environments [1].
To implement an augmented reality system, we must solve
some problems. When virtual objects should be superimposed on the place as if they really exist in the real world,
the geometric and photometric registration problems are especially important. The photometric registration problem
requires the consistency between light sources in the virtualworld and those in the real world.
One of the major approaches to the geometric registration between the real and virtual worlds is a vision-based
method. The methods, which are sometimes referred to as
vision-based tracking or registration, estimate the positionand orientation of user’s viewpoint from images captured
by a camera attached at the user’s viewpoint. Some methods use square markers[2].
On the other hand, to resolve the photometric problem,the positions of light sources in the real world should be estimated in real time. One of the methods acquires a lighting
environment by photographing a mirror ball[3]. The othermethod estimates the illumination distribution in the realworld using cast shadow regions[4]. However, this methodcannot estimate the lighting environment in real time.
This paper describes an augmented reality system which

resolves both the geometric and photometric registration
problems using a 3D marker consisting of a 2D square
marker and a mirror ball, which is shown in Figure 1. We
estimate a simple light source map observing a mirror ball.

2. Geometric and Photometric Registration
2.1. Outline of the proposed method
Figure 2 illustrates the outline of the proposed method.
First, a camera captures a 3D marker which is composed ofa 2D square marker and a mirror ball and is placed in thereal scene. The relationship between the 2D marker and themirror ball and their size are known. Second, the relationship between the marker and camera coordinate system isestimated by detecting the region of the 2D square marker
using an existing method[2]. Using the result, the mirror
ball in the image is detected. Next, a simple light source
map which represents a distribution of light sources in thereal scene is estimated from the mirror ball image. Finally, avirtual object is rendered by using the estimated light sourcemap.
The rest of the section describes the geometric registration and the estimation of light source map in some moredetail.

2.2. Geometric registration
In the geometric registration process, a 2D square region
is detected based on the size, color and shape of the maker.
The model-view matrix, which represents the relationshipbetween the marker and the camera, is estimated by identifying the corners of the markers[2].

Figure 1. A 3D Marker

Square marker
Captured image
CameraMirror ball(a) Region of mirror ball in two views

Virtual object(A) Geometricregistration
Composed image(B) Estimation oflighting condition

Computer(b) Light source map in two views
Figure 2. Flow diagram of the method

2.3. Estimation of lighting environment
The region of mirror ball in the captured image canbe easily determined by using the model-view matrix estimated in Section 2.1 and the relationship between the 2D
square marker and the mirror ball. The brightness of a lightsource in the real world is determined by intensities of pixels in the region of the mirror ball. Next, the directions oflight sources are estimated by using the camera pose andsurface normals of mirror ball points corresponding to thedetected pixels.

3. Experiment
In the prototype system, an IEEE1394 camera (SONY:DFW-VL500) captures a 3D marker placed in the realworld. The captured image is fed into a desktop PC (CPUP4 2GHz, Memory 256MB), and a composed image is generated.
Figure 3 shows experimental results. Figure 3(a) illustrates a region of mirror ball in the captured images, thebrightness of light source is estimated from the intensity ofpixels in the region. In the experiment, the size of the mirror ball region is about 20 2 20 pixels. Figure 3(b) shows
an estimated light source map in two different views. The
size of sphere represents the brightness of each light source.
Figure 3(c) shows a composed image in which a virtual object (teapot) is merged into the real world with geometricand photometric consistency. In the experiment, the frame
rate is 15 frames per second.

4. Conclusion
This paper have described an augmented reality system
which resolves both the geometric and photometric registration problems using a 3D marker composed of a 2D square
marker and a mirror ball. The proposed method is characterized by the estimation of a light source map in real time.

(c) Composed image
Figure 3. Experimental results
Acknowledgment This research is partially supported by Core
Research for Evolutional Science and Technology (CREST) Program ”Advanced Media Technology for Everyday Living” ofJapan Science and Technology Corporation(JST) and also by the
Grant-in-Aid for Scientific Research from the Ministry of Education, Culture, Sports, Science and Technology.

References[1] M. Kanbara, T. Okuma, H. Takemura and N. Yokoya: “AStereoscopic Video See-through Augmented Reality System
Based on Real-time Vision-based Registration,” Proc. IEEE
Virtual Reality 2000, pp. 255–262, 2000.
[2] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto and K.
Tachibana: “Virtual Object Manipulation on a Table-top AR
Environment,” Proc. IEEE/ACM Int. Sympo. on AugmentedReality, pp. 111–119, 2000.
[3] P. Debevec: “Rendering Synthetic Objects into Real Scenes:
Bridging Traditional and Image-based Graphics with Global
Illumination and High Dynamic Range Photography,” Proc.
SIGGRAPH’98, pp. 189–198, 1998.
[4] I. Sato, Y. Sato and K. Ikeuchi: “Illumination Distribution
from Brightness in Shadows: Adaptive Estimation of Illumination Distribution with Unknown Reflectance Properties inShadow Regions,” Proc. ICCV’99, pp. 875–882, 1999.