A Stereoscopic Video See-through Augmented Reality System
Based on Real-time Vision-based Registration
Masayuki Kanbara1 , Takashi Okuma2, Haruo Takemura1 and Naokazu Yokoya1 3
1 Graduate School of Information Science, Nara Institute of Science and Technology

8916-5 Takayama-cho, Ikoma-shi, Nara, 630-0101, JAPANfmasay-ka, takemura, yokoya g @is.aist-nara.ac.jp
2 Electrotechnical Laboratory, MITI
1-1-4 Umezono, Tsukuba-shi, Ibaraki, 305-8568, JAPANokuma@etl.go.jp
3 Nara Research Center, Telecommunications Advancement Organization of Japan
8916-19 Takayama-cho, Ikoma-shi, Nara, 630-0101, JAPAN

Abstract
In an augmented reality system, it is required to obtain
the position and orientation of the user’s viewpoint to display the composed image maintaining correct registrationof real and virtual worlds. All the procedures must be donein real-time. This paper proposes a method for augmented
reality with a stereo vision sensor and video see-throughhead mounted display. It can synchronize the display-timing
between the virtual and real worlds, so that an alignment
error is reduced. The method calculates camera parameters from three markers in image sequences captured bya pair of stereo cameras mounted on a HMD. In addition,
it estimates depth of real world from a pair of stereo images to generate a composed image maintaining consistentocclusions between real and virtual objects. A region of
depth estimation is efficiently limited by calculating a position of virtual object using camera parameters. Finally,
we have developed a video see-through augmented reality
system which mainly consists of a pair of stereo cameras
mounted on the HMD and a standard graphics workstation.
The feasibility of the system has been successfully demonstrated with experiments.

1. Introduction
This paper describes a method of merging real and virtual worlds for a video see-through augmented reality system. Augmented reality produces an environment in which
virtual objects are superimposed on a user’s view of the realenvironment. Augmented reality has received a great dealof attention as a new method for displaying information or
increasing the reality of virtual environments. A number ofapplications have already been proposed and demonstrated[1]. For example, Feiner et al. proposed an augmented re-ality system that explains to an end-user how to maintain aprinter using see-through HMD[2]. State et al. presented
an augmented reality system applied to a medical procedure known as the ultrasound-guided needle biopsy of abreast[3].
To implement an augmented reality system, we must resolve some problems. A geometric registration is especiallythe most important problem because the problem is a principal factor which provides a user with a sense of incongruity. The registration includes a problem of geometricalignment of the real and virtual coordinates and a problem of resolving occlusion between real and virtual objects.
The former problem is considered as one of acquiring theposition and orientation of the user’s viewpoint in terms ofregistering the real and virtual worlds geometrically[4]. The
latter problem can be resolved by measuring the real worldin advance when the real world is static. However, since thereal world is usually dynamic, we must estimate a depth of
the real scene in real-time.
In addition to the problems above, we need to discuss
how to combine a method of visually merging the real andvirtual worlds and a method of resolving geometric registration because there are some methods of composition thatcannot present correct occlusion. The following briefly reviews the respective methods.
In general, the following two major methods are known
for acquiring the user’s viewpoint in geometric registration.

 A method that uses a 3-D tracker, such as electromagnetic, ultrasonic or mechanical trackers; for example, see [2, 3].

 A method that estimates the user’s viewpoint by using

camera images captured at the user’s viewpoint; forexample, see [5, 6, 7]. This is sometimes referred to asa vision sensor.
The 3-D trackers used in the former method can directly
acquire 3-D position and orientation of receivers. However,the drawbacks of the method are that the system requires a
special equipment and its measurement range is limited toa relatively narrow area. On the other hand, the latter canestimate the position and orientation of the user’s viewpointfrom an acquired image and there is potentially no limitation in measurement range. When the relationship betweena camera position and the user’s viewpoint is known, theuser’s viewpoint can be obtained by calculating the camera
parameters from captured images. This means that the traditional techniques studied in the field of computer vision
can be used to measure the viewpoint [8].
There exist two methods of showing a user images inwhich the real and virtual environments are merged.

 An optical see-through shows a user the real environ-ment through a half-transparent mirror and the virtualenvironment reflected on the half-transparent mirror.

 A video see-through shows a user image sequences of
the real environment captured by a camera and the virtual environment overlaid upon the image sequences.

The optical see-through system can show a user the real environment without delay. However, virtual objects are notsynchronized with the real world imagery. This may cause
the displacement between real and virtual objects. This typeof system has another drawback that the amount of lightfrom the real environment is reduced, because it uses a halftransparent mirror. In contrast, the video see-through system makes the display-timing of the real environment andthat of the virtual environment synchronous. Therefore, it
reduces the alignment error. However, the augmented realenvironment is shown to the user with delay [9].
We focus on an augmented reality system that adopts the
combination of vision sensor and video see-through system.
Since the image captured by a camera is used both to estimate camera parameters and to show the user the real environment, the combination is able to synchronize the real andvirtual environments and reduce alignment error between
them. For implementation of an augmented reality systemwith highly realistic sensations, all the processes from acquisition of the user’s viewpoint to displaying the composed
image by using obtained viewpoint must be done in realtime.
This paper is structured as follows. Section 2 briefly reviews related work. Section 3 describes the algorithms tocalculate camera parameters and to estimate the depth ofreal world as well as stereo image composition with examples. In Section 4, the experimental results with theproposed methods and the discussion about the prototype
system are described. Finally, Section 5 summarizes thepresent work.

2. Related WorkThe registration problem is considered as a problem ofacquiring the position and orientation of the user’s viewpoint. Using a vision sensor, if we assume that the optical axes of the cameras are set to be parallel to viewer’s
gaze direction, the acquisition of position and orientationof the user’s viewpoint is equal to the estimation of external camera parameters. A number of methods of estimatingcamera parameters from tracked feature points in the image
sequence have been proposed by several research groups[6, 10, 11, 12]. The authors have already built a prototype
of a vision-based video see-through augmented reality system with a single camera that uses four colored markers asfeature points [5]. In this system, the position and orientation of the user’s viewpoint are calculated from the positions of the feature points in the captured image. However, the monocular vision system does not show the user
the stereoscopic view. Moreover, virtual objects that should
be behind the real objects occlude the real objects, so thatthe occlusion conflict occurs. State, et al. [13] proposed avideo see-through augmented reality system that provides
the user with the stereoscopic view using a pair of stereo
cameras, but the system could not show the user the imagesof virtual objects with a dynamic real world.
This paper describes algorithms to realize a stereo-vision
based augmented reality. The camera parameters can be estimated from captured stereo images containing three features in the scene. In addition, a depth of the real world canbe estimated by a stereo vision technique. Static real objects can be properly treated by the state of the art of stereo
vision techniques in video see-through augmented reality
applications using standard workstations[14, 15]. However,dynamically moving arbitrary objects are still difficult to bemixed with CG objects in real-time without special hardwares [7, 16, 17], maintaining correct occlusions among
them. The method proposed in the paper overcomes thisproblem by estimating depth only for regions where virtual
objects are composed; that is, it is an augmented realityoriented approach to stereo vision.

3. Algorithms for Stereoscopic Video Seethrough Augmented Reality
In order to evaluate the feasibility of the augmented reality environment that uses a pair of stereo cameras, the prototype system of augmented reality was implemented. The
system estimates camera parameters using a pair of stereo
images. The system also realizes the occlusion of virtual
objects by real objects.
Figure 1 shows the flowchart of the prototype system. A
pair of stereo image sequences captured by stereo cameras

left image

A. camera parameter
estimation
tracking
trackingof markers
of markers

right image

input

determination of 3D
position of markers

stereo cameras

calculation of model-view matrix

B. depth estimation
limitation of depth limitation of depth
estimation region estimation region

HMD
outputleft image

right image
C. composition ofreal and virtual images

depth
computation

depth
computation

C. composition ofreal and virtual images

Figure 1. Flow diagram of stereoscopic video see-through augmented reality system.
are fed into a standard workstation. First, in order to capture the sight of user’s viewing direction, the stereo cameras
are mounted on a HMD. The camera parameter is estimatedusing these image sequences of the real world containing
markers. Next, depth of real world is estimated using boththese image sequences and the camera parameter. Finally,
the stereo images are composed with the computer graphics(CG) images (images of virtual objects) and are outputtedto the HMD that is worn by the user.
In the following sections, the details of the camera parameter estimation, depth estimation of the real world andthe image composition are described.

3.1. Camera parameter estimation
In our prototype system, blue markers are used as featurepoints in a real environment. It should be noted that a pair ofstereo cameras are calibrated in advance by using a standard
technique[18]. The following three steps are executed toestimate camera parameters.
1. From the input images, blue regions are extracted andthe positions of markers in the images are obtained.
2. The markers’ 3-D positions in the camera coordinate
system are estimated by using a stereo matching algorithm.
3. A projection matrix (model-view matrix) that represents the relationship between the real and virtual en-

vironments is calculated using 3-D coordinates of extracted markers.
Let us describe these steps in more detail in the following.
3.1.1. Extraction of markers. The extraction of blue
marker regions from the entire image needs considerable
amount of computation. Therefore, only in the first frame of
the image sequence, the blue region extraction is performed
over the entire image. In the subsequent frames, extracted
regions are tracked by using the results obtained in the previous frame. The extracted and tracked regions are used to
calculate the screen coordinate of each marker.
In the first frame of the image sequence, the following
steps are used to extract blue marker regions.
1. The entire images in the first stereo pair are scanned to
extract blue regions.
2. The center of gravity of each blue region is treated as
the screen coordinate of the marker.
3. When three markers are found in both left and right images, stereo matching is performed based on the epipolar constraint. All markers are then labeled (Labels:
1,2,3).
When the processing speed is fast enough, the markers’ positions in adjacent frames can be assumed to be close to eachother. Thus the following steps are used in the subsequent
frames in the image sequence.

1. A search area for each marker is determined in the current frame based on the position of the marker in theprevious frame.
2. The center of gravity for all blue pixels in the searcharea is calculated and is treated as the screen coordinate of the marker. The matching between left andright images in the first frame is used to determine
the stereo pair of markers between the current left andright images.

X

Y

Z

=
=
=

B (xl + xr )

2(xl

xr )

B (yl + yr )

2(xl

xr )

fB
xl

xr

;

;

;

where f is the focal length and B is the baseline length.
Thus when f and B are known, P (X; Y; Z ) is calculated
from the coordinates Pl (xl ; yl ) and Pr (xr ; yr ).
3.1.3. Calculation of model-view matrix. In order to compose the image of a real environment and CG objects, A
model-view matrix that represents the transformation fromthe world coordinate, in which the shape of CG objects isdefined, to the camera coordinate is required. Figure 3 illustrates the relationship between the world and camera coordinates as well as markers. Among a model-view matrix- M, a position of a point in the world coordinate system w, and its position in the camera coordinate system - c, thefollowing equation stands.

c Mw=

Pl (xl ,yl )

Y

right
image

Z

centerof left lensB

cameraX
coordinate system

Pr (x r, yr )fcenter of right lensFigure 2. Geometry of stereoscopic projection of a marker.

2
M = 64

3
T 75

R
0

0

0

:

1

In the prototype system, the world coordinate system issimply defined as follows: The origin is set at the marker labeled as No. 1 (M1 in
Figure 3).

 The -axis is set on the line that connects the markers
xlabeled as No. 1 and No. 2 (M2 in Figure 3).

 The

x
y plane is set on the plane on which threemarkers reside.

According to the definition above, the translation component T can be given by the camera coordinate position of
the marker No. 1. The rotation component R can be calculated with the following steps.
1. When the positions of the markers Nos. 1, 2, and 3
are given as 1 ; 2 , and 3 , respectively, the direction of the vector of each axis of the world coordinate( n ; n ; n) can be defined as follows.

V V

x y z

xn
yn
zn

:

The matrix M is a rigid transformation in homogenous
coordinates consisting of a rotation R and a translation T as
follows.

P(X,Y,Z)f

When the stereo matching of markers or tracking ofmarkers fails, the system starts over the procedure for the
first frame described above.
3.1.2. Determination of 3-D position of markers. In thisphase of the process, the system determines the 3-D position of each marker. The relationship between two cameras
and a marker is illustrated in Figure 2. The origin of thecamera coordinate system is placed at the middle between
the centers of projection of two cameras. The X -axis isset along the baseline of the cameras. The Z -axis is set tothe direction parallel to the optical axes of the cameras. A
marker at P (X; Y; Z ) in 3-D space is projected onto the left
and right images at Pl (xl ; yl ) and Pr (xr ; yr ) in the screen
coordinates, respectively. Then following equations stand:marker

left
image

=
=
=

V

V2 V1
V3 V1 xn  xVn 3 xn V1 xn
xn  y n
;

(

(

)

x y

z

)

;

:

x y

z

2. Normalize n ; n ; and n into 0n ; n0 ; and 0n , respectively.

xn kxxnn k yn kyynnk zn kzznnk
0

=

;

0

=

;

0

=

:

Y

camera coordinate
systemregion of depth
estimation

m1,m2,m3: markers

Z

V3

m3

X

V1

z’n

left image plane

V2

y’n

m2

x’n

m1

world coordinate systemcameracoordinate systemmodel-view
matrix M

Figure 3. Relationship between the world andcamera coordinates.

x yz

3. R can be determined by using 0n ; n0 ; and 0n as follows.

R xn yn zn
=[

0

0

0

CG objectcenter ofleft lens]:

By using T and R, the model-view matrix M can be
uniquely determined. Now it should be noted that the geometric alignment of the real and virtual world coordinatesis achieved by the model-view matrix.

3.2. Depth estimation of real world
In this section, the depth estimation of real world is described. The depth estimation is used to compose the imageof the virtual objects and the image of the real world without
occlusion conflicts. In order to avoid occlusion conflicts, weneed the depth information of only the region onto which
virtual objects are projected. When the positions of virtual
objects are known, the region for depth estimation can belimited as shown in Figure 4. Therefore, the depth estimation region of real world can be determined by projecting
a CG object’s bounding box using the model-view matrix
estimated in Section 3.1. The following steps describe amethod of depth estimation of real world.
1. A position of the CG object in the world coordinate
system is transformed into its position in the camera
coordinate system using the estimated model-view matrix M.
2. A bounding box of the virtual object is projected onto
the left image as shown in Figure 4. A depth estimation region is obtained as a bounding rectangle of the
projected box. The same process is done for the rightimage as well.
3. By adopting the Sobel filter, edges are detected in theregion of depth estimation on the left and right images.

bounding box

center ofright lensFigure 4. Determination of depth estimation
region.

4. Stereo matching is performed and the depth value iscomputed. Note that only pixels on detected edges in
the left image are matched to those in the right image,
matching window size is 5  5 pixels, and similarity
measure is the sum of absolute differences (SAD). In
the same way, the right image as a reference image ismatched to the left image.
5. Matching errors are excluded by considering the consistency between left-to-right and right-to-left matchings. The depth values at the pixels between the edges
are interpolated.

3.3. Composition of real and CG images
By using the estimated model-view matrix and the depth
map of the real world, CG images of virtual objects are
mixed into the image of real world. At this stage, the depthof real world and each virtual object are compared. Whenthe real objects are closer to the user’s viewpoint, a transparent virtual object is drawn at the 3-D position where thereal objects exist. By using a hardware z buffering algorithm, the virtual objects that are farther than the transparent
objects are not actually drawn on the frame buffer. Therefore the composed image is looked as if the real objects are
occluding the virtual objects. These rendering steps are illustrated with examples in Figure 5.
First, only the background image of a real scene is rendered on the frame buffer as in Figure 5 (a). Z-buffer value
is also set to the farthest value through out the screen. Then,
Z-buffer of the real objects are set to the depth of the real
objects in Figure 5 (c). The regions of real objects are illustrated with gray levels of depth in Figure 5 (b). Finally, thevirtual objects are rendered by using the model-view matrix in Figure 5 (d),(e). These steps are applied to both left

(b) depth estimation result(d) rendered virtual object(c) transparent object
generated from depth map(a) real scene image(e) composed image

Figure 5. Process of image composition.
stereo cameras

IK-UM42

input

mediamask

outputHMD

DIVO

Onyx2 IRgraphics board

graphics workstation

Figure 7. Hardware configuration of prototype
system.
Figure 6. Appearance of stereoscopic video
see-through HMD.
and right images for obtaining a stereo pair of composite
images.

4. Experiments
4.1. Prototype system and results
In the present system, two small identical CCD cameras (Toshiba IK-UM42) are mounted on a HMD (Olympus Media Mask) as shown in Figure 6. An optical seethrough function of the HMD is not used; that is, it works inclosed-view mode. The baseline length between two cameras is set to 6.5 cm. The optical axes of the cameras are
set to be parallel to the viewer’s gaze direction (actually
the head direction). The images captured by the cameras
are fed into a graphics workstation (SGI Onyx2 IR: 16CPU

MIPS R10000 195MHz) through the digital video interface(DIVO). The input real world images are merged with virtual objects and outputted from the DIVO interface to the
HMD. The hardware configuration of the whole system is
illustrated in Figure 7.
Figure 8 shows the result of the stereoscopic image composition. The images on the first and second rows (Figure 8(a),(b)) show the composition of a virtual sake cask
into a real desktop scene by using three markers placed on
the desktop. As is observed from the Figure 8, the stereoscopic composition of the virtual sake cask and the real environment are realized. In the third and fourth rows (Figure8(c),(d)), real objects are placed in front of or back of thevirtual sake cask. As we can see from these examples, theocclusion between the virtual and real objects are correctly
established. Moreover, the stereoscopic video see-throughHMD gives a good depth sensation to users and has improved the quality of augmented reality.

(a)

(b)

(c)

(d)

Left image

Fuse
wall-eyed

Right image

Fusecross-eyed

Left image

Figure 8. Composed stereo images.

4.2. Discussion
The average frame rate of the system is 10 frames per
second, when the virtual object shown in Figure 8 (about
3700 polygons with a texture) is synthesized. This meansthat 100 ms are allocated for each frame. The approximate
processing time at each stage is as follows: 20 ms for camera parameter estimation, 40 ms for depth estimation of realworld, 30 ms for image composition with CG. Due to thenature of video see-through augmented reality, there is no
synchronization error between the real scene and virtual objects. On the other hand, an entire scene has a time latencyof 133 ms.
If the system cannot locate at least three markers in thereal environment, it cannot estimate the model-view matrix.

To overcome this problem, we should set more markers in
the real environment, so that the system could stably estimate the model-view matrix.
The vision sensor essentially has no limitations in measurement range. However, the present prototype system actually has a limitation in measurement range because thesystem uses a limited number of predefined markers. If
we could automatically detect and track feature points otherthan markers in a real scene to calculate the model-view matrix, a measurement range of the system would be extended.

5. Conclusion
In this paper, we have proposed a method of composing virtual (CG) objects with real world images in real-timefor video see-through augmented reality applications. As apilot study, we also have developed a prototype of stereoscopic video see-through augmented reality system, which
is implemented by using existing computer vision techniques on a graphics workstation and a HMD with a pairof CCD cameras. The prototype system can produce composite images of real and virtual objects nearly at video-rate,
maintaining correct occlusions between CG objects and real
objects. It should be noted that the system can provide auser with excellent 3-D depth sensation of augmented environments. We have proven that the stereo-vision based
video see-through augmented reality system is feasible andhas the possibility of building up actual applications.
In the future work, we will concentrate our attention on
robust estimation of the model-view matrix in some ways;
for example, (1) using more than three markers, (2) automatic detection and tracking of natural feature points in areal scene without predefined markers.
Acknowledgments
This work was supported in part by Grant-in-Aid for Scientific Research under Grant No. 09480068 from the Ministryof Education, Science, Sports, and Culture, and also by the
Telecommunications Advancement Organization of Japan.

References[1] R. T. Azuma: “A Survey of Augmented Reality,” Presence,Vol. 6, No. 4, pp. 355–385, 1997.
[2] S. Feiner, B. Maclntyre and D. Seligmann: “Knowledgebased Augmented Reality,” Commun. of the ACM, Vol. 36,
No. 7, pp. 52–62, 1993.
[3] M. Bajura, H. Fuchs and R. Ohbuchi: “Merging Virtual Objects with the Real World: Seeing Ultrasound Imagery within
the Patient,” Proc. SIGGRAPH’92, pp. 203–210, 1992.
[4] M. Tuceryan, D. S. Greer, R. T. Whitaker, D. E. Breen, C.
Crampton, E. Rose and K. Ahlers: “Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System,” Trans. on Visualization and computer graphics,
Vol. 1, No. 3, pp. 255–273, 1995.
[5] T. Okuma, K. Kiyokawa, H. Takemura and N. Yokoya: “AnAugmented Reality System Using a Real-time Vision Based
Registration,” Proc. ICPR’98, Vol. 2, pp. 1226–1229, 1998.
[6] M. Uenohara and T. Kanade: “Vision-Based Object Registration for Real-time Image Overlay,” Proc. CVRMed’95,
pp. 13–22, 1995.
[7] N. Yokoya, H. Takemura, T. Okuma and M. Kanbara:“Stereo Vision Based Video See-through Mixed Reality,” in
Mixed Reality - Merging Real and Virtual Worlds, OhmshaSpringer Verlag, pp. 131–145, 1999.
[8] G. J. Klinker, K. H. Ahlers, D. E. Breen, P-Y. Chevalier, C.
Crampton, D. S. Greer, D. Koller, A. Kramer, E. Rose, M.

Tuceryan and R. T. Whitaker: “Confluence of Computer Vision and Interactive Graphics for Augmented Reality,” Precense, Vol. 6, No. 4, pp. 433–451, 1997.
[9] E. K. Edwards, J. P. Rolland and K. P. Keller: “Video Seethrough Design for Merging of Real and Virtual Environments,” Proc. VRAIS’93, pp. 197–204, 1993.
[10] M. Bujura and U. Neumann: “Dynamic Registration Correction in Video-Based Augmented Reality Systems,” IEEE
Computer Graphics and Applications, Vol. 15, No. 5, pp. 52–
60, 1995.
[11] J. Rekimoto: “Matrix: A Realitime Object Identification andRegistration Method for Augmented Reality,” Proc. APCHI,
pp. 63–68, 1998.
[12] A. State, G. Hirota, D. T. Chen, W. F. Garrett and A. Livingston: “Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking,” Proc.
SIGGRAPH’96, pp. 429–438, 1996.
[13] A. State, A. Livingston, W. F. Garrett, G. Hirota and H.
Fuchs: “Technologies for Augmented Reality Systems: Realizing Ultrasound-Guided Needle Biopsies,” Proc. SIGGRAPH’96, pp. 439–446, 1996.
[14] M. M. Wloka and B. G. Anderson: “Resolving Occlusion in
Augmented Reality,” Proc. 1995 ACM Sympo. on Interactive
3D Graphics, pp. 5–12, 1995.
[15] B. Ross: “A Practical Stereo Vision System,” Proc.
CVPR’93, pp. 148–153, 1993.
[16] T. Kanade, A. Yoshida, K. Oda, H. Kano and M. Tanaka:“Stereo Machine for Video-rate Dense Depth Mapping and
Its New Application,” Proc. CVPR’96, pp. 196–202, 1996.
[17] N. Yokoya, T. Shakunaga and M. Kanbara: “Passive Range
Sensing Techniques: Depth from Images,” IEICE Trans. Information and Systems, Vol. E82-D, No. 3, pp. 523–533,
1999.
[18] R. Y. Tsai: “A Versatile Camera Calibration Techniquefor High-accuracy 3D Machine Vision Metrology Using
Off-the-Shelf TV Cameras and Lenses,” IEEE Journal of
Robotics and Automation, Vol. RA-3, No. 4, pp. 323–344,
1987.