2008 International Workshop on Ubiquitous Virtual Reality

Real-time Geometric Registrationof Real and Virtual Worlds
Using a Feature Landmark Database with Priorities

Takafumi Taketomi, Tomokazu Sato,
Sei Ikeda and Naokazu Yokoya
Graduate School of Information Science
Nara Institute of Science and Technology, Japan

Augmented Reality

2Camera positions andpostures are required

Real World

GeometricRegistration

Mixed real and virtual worlds

Virtual World

Requirements for camera parameter estimation
 Applicable to various environments
 Absolute position and posture can be estimated Work in real-timeRelated Work in Geometric Registration

3z Sensor based method{Geometric registration in pixel order is difficult.

z Model based method

3D ModelKey frame{Application for complex environments is difficult.

z Feature point based method［Vacchetti et al. 2004］{Application for complex environments is easy.
Correspondence between
landmarks and feature points
Landmark database

Input image［Oe et al. 2005］［Klein et al. 2007］

Visual SLAM based method
Can not estimate absolute
camera position and posture

Feature landmark based method
Can not work in real-time

Flow of Our Previous Method (Oe et al. 2005)
Database construction(Offline process)4

Camera parameter estimation(Online process)Detection offeature points
Omni-directional
Video

Reconstructionof environment

Acquisition of landmark information

Landmark Database

Selection oflandmarks

Input image

Matching

Camera parameter estimation

Flow of Our Previous Method (Oe et al. 2005)
Database construction(Offline process)5

Camera parameter estimation(Online process)Detection offeature points

High computational cost
Omni-directional
Video

Reconstructionof environment

Acquisition of landmark information

Landmark Database

Selection oflandmarks

Input image

Matching

Camera parameter estimation

Goal and Approach
Goal
Real-time geometric registration using
feature landmark database
Approach
Reduction of matching pairs :• Reduction of feature points byestimating tentative camera parameter• Reduction of landmarks by selecting
landmarks with high priorities

6Registration Using Landmark Database

7

Camera parameter estimation(Online process)

Database construction(Offline process)

Reduction offeature points
Detection offeature points
Omni-directional
Video

Input image

Reconstructionof environment

Matching

Acquisition of landmark information

Landmark DatabaseSelection oflandmarks

Camera parameter estimation

Feature Selection

8

Range of search is usually determined by camera parametersof the previous frame.
Projected position of landmark
Range of search

Current frame

Camera position and posture of theprevious frame

ProblemRange of search is very large.

A lot of feature points are tested
as matching candidates.

Reduction of Feature Points

9

Reduction of feature points by estimating
tentative camera parameters.

Previous frame

Current frame

Camera position and posture of the previous frame
Position of landmarks in the previous frameReduction of Feature Points

10

Reduction of feature points by estimating
tentative camera parameters.

Previous frame

Current frame

Tentative camera position and posture
Position of landmarks in the previous frame
Position of landmarks in the current frame
Projected position of landmark

Range of search becomes smaller than that in the previous method.

The number of feature points is reduced

Registration Using Landmark Database
Database construction(Offline process)Camera parameter estimation(Online process)

Reduction oflandmarks
Omni-directionalVideo

Reconstructionof environment

Acquisition of landmark information

Landmark Database

Selection oflandmarks

11

Detection offeature points

Input image

Matching

Camera parameter estimation

Landmark Selection

12

Landmarks are selected based ongeometric information.
Landmark database

Landmarks visible from the previous
camera position and posture

Camera position and postureof the previous frame
Problem:

Ineffective landmarks are often selectedfrom the database.

Calculation of Priorities of Landmarks

13

Priorities are determined by calculating probabilities thatlandmarks are used in camera parameter estimation.
Ei：Frequency that the landmark i is determined
as an inlier by robust estimation.
Si：Frequency that the landmark i isselected from the database.

Pi = Ei / S i

Frequency information

Landmark database

UserPriorities are updated by estimation results

Experiments

14

The proposed method is compared with the previous
method [Oe et al.05] with respect to the robustness
and the computational cost.
Experimental conditions
Previous method
Input image

Proposed method

Resolution 720x480 pixels，15 fps，1000 frames

CameraSONY DSR-PD-150

PCCore 2 Extreme 2.93GHz，Memory 2GB

Image template sizeof landmarks

15×15 pixels

Range of searchin tracking process-

120×60 pixels

Range of searchin matching process

120×60 pixels

20×20 pixels

Learning data-

Estimation resultsof three sequences

Initial value of priorities-0.5

Evaluation of the Robustness
m15-57 -54 -51 -48 -45 -42 -39 -36 -33 -30 -27 -24 -21 -18 -15 -12 -9-6

0-3-3 0

3

6

912-6
Previous method
Proposed method
Ground truth-9-12mEstimated camera path and the ground truth※ The ground truth is created by manually corresponding between landmarks
and features.
※ Number of landmarks selected from database: 100 per frame

Comparison of accuracy
Previous method

Proposed method

Error in position (mm)

220

226

Error in optical axis (degree)

0.37

0.41

Evaluation of the Robustness

160-21-18-15-12-9-3

Previous method
Proposed method
Ground truth

Previous method-6-9Proposed method

400
300
200
1000

Previous method
Proposed method

N u m b e r o f fa ilu refra m e s

E r r o r o f p o s it io n(m m )Reduction of Landmarks by Using Priorities

100 90 80 70 60 50 40 30 20Number of landmarks N

1517

Previous method
Proposed method

1050100 9080 70 60 50 40
Number of landmarks N3020

Comparison of accuracy
Previous method Proposed method(N=80)(N=60)
Error in position (mm)

235

231

Error in the optical axis (degree)

0.39

0.42

Improvement of Computational Cost
processing time (ms)18

Ratio ofprocessing time

Previous method(N=80)

Proposed method(N=60)

Estimation oftentative camera parameter-28-

Landmark selection

1210.08

Matching

316

150.05

Camera parameter estimation

61

170.28Other processing

451.25

total

393(2.5fps)

66(15.1fps)

0.16

The proposed method can work in real-time.

Result of Registration

19

Number of landmarks selected from database: 60
Processing time：66 milliseconds per frame

Summary20• We propose real-time camera parameter estimation methodby reducing matching pairs of feature points and landmarks.
– Reduction of feature points by estimating tentative camera parameter– Reduction of landmarks by selecting high prior landmarks• Camera parameter can be estimated in large and naturalenvironments.

Feature Work• Verification in various environments• Automatic initial camera parameter estimation

21Thank you!!
z Discussions & More information• NAIST, 8916-5 Takayama, Ikoma, NARA 630-0192 JAPAN• mailto: takafumi-t@is.naist.jp• Web: http://yokoya.naist.jp/index.html