Real-time Estimation of Light Source Environmentfor Photorealistic Augmented Reality
Masayuki Kanbara and Naokazu Yokoya
Graduate School of Information Science, Nara Institute of Science and Technology
8916-5 Takayama-cho, Ikoma-shi, Nara 630-0192, JAPAN{kanbara, yokoya } @is.naist.jp

Abstract
This paper proposes a vision-based augmented reality
system with correct representation of attached and castshadows. To realize a seamless augmented reality system,
we need to resolve a number of problems. Especially, thegeometric and photometric registration problems are important. These problems require to estimate the positionsof light sources and user’s viewpoint. The proposed system
resolves the problems using a 3D marker which combines a
2D square marker and a mirror ball. The 2D marker andthe mirror ball are used to estimate the relationship between
the real and virtual worlds and the positions of light sourcesin the real world, respectively.

1. Introduction
Augmented reality produces an environment in which
virtual objects are superimposed seamlessly on user’s viewof the real environment. Augmented reality has received
a great deal of attention as a new method for displaying
location-based information or increasing the reality of virtual environments [1, 2, 3]. To implement an augmented
reality system, we must solve a number of problems. Whenvirtual objects should be superimposed on a certain place
as if they really exist in the real world, the geometric andphotometric registration problems are especially important.
The geometric registration is especially the most important problem because the problem is a principal factor which
provides a user with a sense of incongruity. The registration includes a problem of geometric alignment of the realand virtual coordinates in real time. The problem is considered as one of acquiring the position and orientation of theuser’s viewpoint in a world coordinate system. One of themajor approaches to the geometric registration between thereal and virtual worlds is a vision-based method [4, 5]. The
methods, which are sometimes referred to as vision-based
tracking or registration, estimate the position and orientation of user’s viewpoint from images captured by a camera
attached at the user’s viewpoint. Some methods use squaremarkers[2].

On the other hand, the photometric registration problem
requires the consistency between light sources in the virtualworld and those in the real world. Therefore, to resolve thephotometric problem, the positions of light sources in thereal world should be estimated in real time[6]. Debevec has
acquired a lighting environment by photographing a mirrorball[7]. Another methods estimates the illumination distribution in the real world using cast shadow regions[8]. However, this method cannot estimate the lighting environment
in real time.
Matsuoka has proposed a method which resolves geometric and photometric registration by using a square
marker which is combined with sensors for estimating alighting environment [9]. However, since the photodiodes
sensors are used to estimate the position of light sources,
the hardware configuration of the system is complex.
This paper describes an augmented reality system which
resolves both the geometric and photometric registration
problems using a 3D marker consisting of a 2D square
marker and a mirror ball which is painted black as shownin Figure 1. The 2D marker is used to estimate the relationship between the real and virtual worlds. We estimate
a simple light source map observing a mirror ball. Since itneeds to estimate position, color and intensity of only lightsources, the mirror is painted black to resolve a dynamic
range problem.
This paper is structured as follows. Section 2 describes
the algorithms to calculate camera parameters and to estimate the position of light sources in real world as well as anFigure 1. A 3D Marker.

Marker region detection

The rest of the section describes the geometric registration and the estimation of light source map in some moredetail.

Detection of corner points

2.2. Geometric registration

Geometric registration

Square Marker

Calculation of a model-view matrix
Captured image

Photometric registration
Detection of mirror ball region
Estimation of light source map

In the geometric registration process, a 2D square region
is detected based on the size, color and shape of the maker.
The model-view matrix, which represents the relationshipbetween the marker and the camera, is estimated by identifying the corners of the markers[2]. In addition, to detect
the marker region stably, a threshold of the binarization process is changed dynamically.
Step1 Marker region detection

Mirror ball

Generation of composed images

Figure 2. Flow diagram of the registration
method.
image composition method. In Section 3, the experimental
results with a proposed method and the discussion about theprototype system are described. Finally, Section 4 summarizes the present work.

First, candidate regions of the square marker are detected by using standard binarization and segmentation techniques from the captured image. When themultiple regions are detected from the captured image,the biggest region is selected as the square marker region. To detect the marker for changing the light environment, a threshold of the process of binarization
is determined by a mean intensity value of the marker
region in the previous frame.
Step2 Detection of corner points

2. Geometric and Photometric Registration
2.1. Outline of the proposed method
Figure 2 illustrates the outline of the proposed method.
First, a camera captures a 3D marker which is composed ofa 2D square marker and a mirror ball and is placed in thereal scene. The relationship between the 2D marker and themirror ball and their size are known. The method consistsof three processes. In the following, each process is briefly
described.
1. Geometric registration
The relationship between the marker and the camera
coordinate system is estimated by detecting the regionof the 2D square marker using an existing method.
2. Photometric registration
Using the result of the first process, a region of the mirror ball in the image is detected. To avoid the dynamic
range problem, the mirror ball is painted black. Next,a simple light source map which represents a distribution of light sources in the real scene is estimated fromthe region of the mirror ball.
3. Generation of composed images
Finally, a virtual object is rendered with correct attached and cast shadows by using the estimated lightsource map.

The four points of marker’s corners are detected fromresults of segmentation process in Step 1.
Step3 Calculation of a model-view matrix
A model-view matrix, which represents relationshipbetween the real and virtual coordinate systems, is calculated by using the positions of the four points ofmarker’s corners as well as [2].

2.3. Photometric registration: Estimationof lighting environment
The region of the mirror ball in the captured image canbe easily determined by using the model-view matrix estimated in Section 2.2 and the known relationship between
the 2D square marker and the mirror ball. Next, the directions of light sources are estimated by using the camera poseand surface normals of the mirror ball points corresponding
to the detected pixels as shown in Figure 3. The directionVr of light source can be calculated by using the viewing
vector Vray and the normal vector VN of the mirror ballpoint as follows:
Vr = −2(VN · Vray )VN + Vray .

(1)

In this method, to decrease calculation cost, all viewing
vectors are approximately set to the direction parallel to theoptical axis of the camera. When the size of the region ofmirror ball is assumed to be 20 × 20 pixels, directions ofReflection vector

VrNormal vector

VN

Mirror ball

Image plane

VrayViewing vector

Viewpoint

Figure 3. Relationship between a viewpoint
and a light source.

(CPU P4 2GHz, Memory 256MB), and a composed image
is generated.
Figure 5 (a) shows an input image. Figure 5 (b) shows acomposed image in which a virtual object (teapot) is merged
into the real world with only attached shadow. Figure 5(c) shows a composed image in which the virtual object ismerged with both attached and cast shadows. In Figure 5(c), we can see that the virtual object really exists in realworld. In the experiment, the frame rate is 20 frames per
second. Figure 6 shows composed images with user’s varying viewpoints. Figure 6(a) illustrates a region of mirror
ball in the captured images,The system can provide a user with a photorealistic augmented reality environment in the same lighting environment as the real scene. However, since the region size ofmirror ball in the captured image is limited in the present
experimented setup, the directions of light sources are estimated sparsely. Therefore, cast shadows are rendered withdiscontinuities, which can be observed in Figure 6.

4. Conclusion

Figure 4. An example of light source map.
the light sources are discretized as shown in Figure 4. The
brightness and color of a light source in the real world aredetermined by intensities and color of pixels in the regionof the mirror ball, respectively.

2.4. Generation of composed images
By using the estimated model-view matrix and the lightsource map of the real world, CG images of virtual objects
are mixed into the image of real world. First, a virtual object
is registered geometrically by using the model-view matrix.
Next, cast and attached shadows are rendered by the lightsource map. In the rendering process of attached shadows, alimited number of light sources are used in order to achieve
real-time rendering (the brightest eight light sources are actually used in experiments). On the other hand, cast shadows are rendered using all the light sources that are obtainedin the light source map assuming that the marker is put on aplane.

3. Experiment
We have developed a prototype of video see-throughaugmented reality system, for which an IEEE1394 camera(SONY: DFW-VL500) captures a 3D marker placed in thereal world. The captured image is fed into a desktop PC

This paper has described an augmented reality system
which resolves both the geometric and photometric registration problems using a 3D marker composed of a 2D square
marker and a mirror ball. The proposed method is mainly
characterized by the estimation of a light source map in realtime. The geometric registration problem is resolved byusing an image of a square marker captured by a camera
which is attached near user’s viewpoint. The light source
map is estimated by analysing a mirror ball region which
is detected by using the result of geometric registration.
Thereby, the proposed method can realize augmented reality environments which contain virtual objects in the samelighting environment as the real scene. As a pilot study, we
also have developed a prototype of video see-through augmented reality system, which has resolved geometric andphotometric registration at the same time. The prototype
system can produce composite images of real and virtualobjects nearly at video-rate, maintaining correct cast andattached shadows.
Acknowledgment This research is partially supported by theGrant-in-Aid for Scientific Research from the Ministry of Education, Culture, Sports, Science and Technology.

References[1] M. Kanbara, T. Okuma, H. Takemura and N. Yokoya: “AStereoscopic Video See-through Augmented Reality System
Based on Real-time Vision-based Registration,” Proc. IEEE
Virtual Reality 2000, pp. 255–262, 2000.
[2] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto and K.
Tachibana: “Virtual Object Manipulation on a Table-top AR
Environment,” Proc. IEEE/ACM Int. Sympo. on AugmentedReality, pp. 111–119, 2000.

(a) Input image(b) Composed imagewith attached shadow(c) Composed image
with attached and cast shadows

Figure 5. Results of geometric and photometric registration.

(a) Region of mirror ball(b) Composed images
Figure 6. Augmented scenes with varying viewpoints and color of light sources.
[3] Y. Yokokohji, Y. Sugawara and T. Yoshikawa: “Accurate Image Overlay on See-through Head-mounted Displays Using
Vision and Accelerometers,” Proc. IEEE Virtual Reality 2000,
pp. 247–254, 2000.

[7] P. Debevec: “Rendering Synthetic Objects into Real Scenes:
Bridging Traditional and Image-based Graphics with Global
Illumination and High Dynamic Range Photography,” Proc.
SIGGRAPH’98, pp. 189–198, 1998.

[4] J. Rekimoto: “Matrix: A Realitime Object Identification andRegistration Method for Augmented Reality,” Proc. APCHI,
pp. 63–68, 1998.

[8] I. Sato, Y. Sato and K. Ikeuchi: “Illumination Distribution
from Brightness in Shadows: Adaptive Estimation of Illumination Distribution with Unknown Reflectance Properties inShadow Regions,” Proc. ICCV’99, pp. 875–882, 1999.

[5] A. State, G. Hirota, D. T. Chen, W. F. Garrett and A. Livingston: “Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking,” Proc.
SIGGRAPH’96, pp. 429–438, 1996.

[9] H. Matsuoka, A. Onozawa and E. Hosoya: “Environment
Mapping for Objects in the Real World: A Trial Using ARToolkit,” Proc. Int. Augmented Reality Toolkit Workshop,
2002.

[6] M. Kanbara and N. Yokoya: “Geometric and PhotometricRegistration for Real-time Augmented Reality,” Proc. ISMAR2002, pp. 279–280, 2002.