Robust 2D-3D alignment based on geometrical consistency
Kenji HARA, Yuuki KABASHIMA, Yumi IWASHITA, Ryo KURAZUME, Tsutomu HASEGAWA
Kyushu University, Japan, yumi@irvs.is.kyushu-u.ac.jp
Abstract
This paper presents a new registration algorithm of a
2D image and a 3D geometrical model, which is robust for
initial registration errors, for reconstructing a realistic 3D
model of indoor scene settings. One of the typical techniques of pose estimation of a 3D model in a 2D image isthe method based on the correspondences between 2D photometrical edges and 3D geometrical edges projected on the
2D image. However, for indoor settings, features extracted
on the 2D image and jump edges of the geometrical model,
which can be extracted robustly, are limited. Therefore, it isdifficult to find corresponding edges between the 2D image
and the 3D model correctly. For this reason, in most cases,
the relative position has to be manually set close to correct
position beforehand. To overcome this problem, in the proposed method, firstly the relative pose is roughly estimatedby utilizing geometrical consistencies of back-projected 2D
photometrical edges on a 3D model. Next, the edge-based
method is applied for the precise pose estimation after theabove estimation procedure is converged. The performanceof the proposed method is successfully demonstrated withsome experiments using simulated models of indoor scene
settings and actual environments measured by range andimage sensors.

1. Introduction
This paper deals with the problem of pose estimation of
a 3D geometric model in a 2D image for creating a realistic 3D model of indoor scene settings. Several registration
techniques of a 2D image and a 3D geometric model havebeen proposed so far. One of the typical techniques is themethod based on the correspondence of 2D photometricaledges and projected 3D geometrical edges on the 2D image. However, for indoor settings, features extracted on the
2D image and jump edges of the geometrical model, which
can be extracted robustly, are limited. Therefore, it is difficult to find corresponding edges between the 2D image andthe 3D model correctly. For this reason, in many cases, therelative position has to be manually set close to correct position beforehand, since it is likely to get stuck into local
minimums if an initial registration error is large.
In this paper, a new registration algorithm of a 2D image and a 3D geometric model is proposed. The proposed
method is able to estimate optimum pose of 3D model in 2D
image robustly against initial registration error. In general,most of urban or indoor scene settings consist of multiple
planar surfaces, and these planar surfaces and their intersection lines have some geometrical relations with other planar
surfaces and intersection lines, respectively. While most of
conventional methods proposed so far utilize 2D photometrical edges and projected 3D geometrical edges on the 2D
image, the proposed method evaluates geometrical consistencies of back-projecting 2D photometrical edges on the
3D model. By utilizing geometrical properties, such as linearity, parallelism, and orthogonality, the relative pose canbe robustly estimated even if an initial registration error islarge.
The proposed method consists of two registration algorithms, one is based on geometrical consistency of artificial 3D scene settings [3, 11] and the other is based oncorrespondences between 2D photometrical and 3D geometrical edges [6]. Firstly, the geometrical consistencybased registration method is applied for determining relative pose robustly against an initial registration error. Next,
for the precise pose estimation, the edge-based registration
method is applied after the geometrical consistency-based
registration method is converged. The performance of the
proposed method is successfully demonstrated with someexperiments using simulated 3D geometrical models of indoor scene settings and actual environments measured byrange and image sensors.

2. Related work
For the precise alignment of 2D images and 3D geometrical models, Kurazume [6] proposed the simultaneous registration algorithm using 2D texture images and reflectance images, which are provided as a side product ofrange images by most of Time-of-Flight range sensors. In
this method, a number of photometrical edges extracted
from a texture and a reflectance images are registered andthe relative pose between them is determined using robust
M-estimator. Epipolar constraints are also utilized to estimate relative poses of multiple texture images simultaneously. Elstrom [4] also used extracted feature points inreflectance images and texture images by a corner detector, and determines correspondence between these featurepoints. Umeda [16] also utilizes the reflectance image, but
this method introduces the optical flow constraint between
the reflectance image and the texture image. Intrinsic andextrinsic parameters are determined using the least squares

method.
Other than the methods using reflectance images, someregistration techniques using silhouette images or contour
lines in 2D image plane have been proposed. Lensch [9],[8], [10] proposed a silhouette-based approach. The size of
XOR regions of silhouette image of a 2D image and a projected 3D model is defined as the similarity measure, andthe optimum pose which minimizes the size of the XOR
regions is determined using the Downhill Simplex method.
Brunie [1] and Lavallee [7] utilize a pre-computed 3D distance map of a free form object for 3D pose estimation. The
error metric is defined as the minimum distance between thesurface of the 3D model and a projection ray, and the sumof the error is minimized using the Levenberg-Marquardt
method. To make the registration process efficiently, 3D
distance from the surface is pre-computed and stored by theoctree structure. Zuffi [17] applies this algorithm for thepose estimation of a knee joint in a single X-ray image fortotal knee replacement surgery. In the contour-based approach, the error is computed as the sum of distances between points on a contour line in a 2D image and on a projected contour line of a 3D model [2], [13], [14]. Iwashita[5] proposed the fast alignment algorithm utilizing a 2D distance map constructed by Level Set Method. However, features extracted from a 2D image or a 3D model are limitedin the indoor settings, and it is likely to get stuck into local
minimums if an initial registration error is large.
Liu [12] proposed a registration algorithm aimed at urban scene objects. This method is based on the correspondences of 3D geometrical edges and 2D photometricaledges, but it is likely to get stuck into local minimums in
the case that an initial registration error is large. In [15],relative positions between image and range sensors are estimated roughly using vanishing points on 2D images, andthe optimum pose is estimated by comparing 2D photometrical edges and 3D geometrical edges projected on the 2D
image. However, to estimate vanishing points stably, plentyof parallel lines have to be extracted from the 2D image.
Therefore, this method is not suitable for urban environments in which enough number of parallel lines can not be
extracted, such as indoor scene settings consisting of a few
number of planes.

3. New 2D-3D registration algorithm using geometrical consistencies
In this section, we propose a new registration algorithmof a 2D image and a 3D geometric model which can estimate optimum pose robustly against initial registration error. Hereafter we assume that a 3D geometric model of ascene is constructed and represented by a number of small
triangular patches. Intrinsic parameters of an image sensor
are also calibrated precisely.
The proposed procedure is summarized as follows:

1. 2D photometrical edges in the 2D image are extracted
by feature detectors such as Canny operator.
2. Straight lines and planar surfaces in the 2D image aredetermined from 2D photometrical edges extracted instep 1.
3. The 3D geometric model is placed at an arbitrary position.
4. Straight lines and planar surfaces are back-projectedto the 3D model using the updated parameters of therelative position.
5. Patches of the 3D model on which the straight lines
and planar surfaces are back-projected are identified
and their 3D positions are calculated by utilizing thehigh speed rendering function of the OpenGL hardware accelerator.
6. The evaluated value of the geometrical consistencies
is calculated and the relative position between the 2D
image and the 3D model is estimated using steepest
descent method or conjugate gradient method.
7. Step 4 ∼ 6 are repeated until the evaluated value converges.
8. The precise pose is estimated using the edge-based
method, which is the method based on the correspondences between 2D photometrical edges and projected
3D geometrical edges on the 2D image.
Step 3 ∼ 7 and Step 8 show the geometrical consistencybased and edge-based registration method, respectively.
The above procedure is explained in more details with someexamples in the following sections.

3.1. Registration method based on geometrical consistencies
3.1.1

Detection of straight lines and planar surfaces.

Straight lines and planar surfaces in the 2D image are extracted from 2D photometrical edges(Fig.1(a)). Firstly,
straight lines are determined with Hough transform or operator’s instruction. Next, to determine planar surfaces, adistance map is constructed, which indicates the distancefrom a pixel in the 2D image to the nearest 2D photometrical edge(Fig.1(b)). By binarizing the distance image with aproper threshold, regions which are far from edges are extracted as planar surfaces(Fig.1(c).
3.1.2 Geometrical consistencies.
In this section, the evaluated value of the geometrical consistencies is explained. The evaluated value indicates thedegree of fitness of the geometrical consistencies defined(a) Extracted photometrical edges(b) Distance image(c) Binary image

Figure 1. Detection of planar surfaces.

Projected straight line

Projected straight line
Straight line

Straight line

for the back-projected 2D edges on the 3D model. Specifically, this value is defined as the linear sum of the following
three evaluated values in this paper.
• Linear consistency• Planar consistency• Parallel and orthogonal consistencies

2D image
Geometric modelThus, the linear consistency is defined as the following
equation.
Elinearity (R, t) =λmin ( i ; R, t)
0≤i<N umsl

λmed ( i ; R, t)

Geometric model(b)

Figure 2. Linear consistency.

Under the relative position between the 2D image and 3D model (R, t), the covariance matrix of the 3D positions of the patches of the
back-projected straight line SLi (0 ≤ i <N umsl , N umsl is the number of detected straightlines.) and eigenvalues of the covariance matrix
are calculated.
Here, three eigenvalues are defined as λmin ( i ; R, t), λmed ( i ; R, t), λmax ( i ; R, t)(0 < λmin ( i ; R, t) < λmed ( i ; R, t) <λmax ( i ; R, t)), and eigenvectors are defined as
Xmin ( i ; R, t), Xmed ( i ; R, t), Xmax ( i ; R, t), respectively.
Xmax ( i ; R, t), Xmed ( i ; R, t), andXmin ( i ; R, t) indicate the directions corresponding
to the maximum, medium, and minimum variances,respectively. Therefore, the smaller the λmin ( i ; R, t)and λmed ( i ; R, t) are, the higher the possibility thatthe back-projected straight line is aligned along a
straight line is.



2D image(a)

The definitions of these consistencies are explained in moredetails.
• Linear consistency
In most cases, a 2D straight line in the 2D image canbe regarded as the projection of a 3D straight line in
the 3D model. Thus, if the 2D image is aligned to the
3D model precisely, the patches of the 3D model on
which the 2D straight line are back-projected shouldbe aligned along a straight line in 3D space. However, if these patches are involved in two different planar surfaces, the back-projected 2D straight line is divided into two straight lines on the 3D model. This isinconsistent with the fact that a 2D straight line corresponds to a single 3D straight line in 3D space (Fig.2).
Therefore, by evaluating the degree of the straightnessof these 3D patches, the correctness of the 2D-3D registration can be estimated. In this paper, we define thiscondition as the linear consistency.

+

View
Point

(1)

0≤i<N umsl

Xmax ( i ; R, t) is the directional vector of the approximated 3D line of the back-projected straight line SLi
in 3D space.

• Planar consistency
In case that the patches on which a planar surface inthe 2D image is back-projected are involved in a single
planar surface, these patches are aligned evenly on the
3D model. However, if these patches are involved in
two different planar surfaces, the back-projected planar surface in the 2D image is divided into two planar
surfaces on the 3D model. This is inconsistent withthe fact that a 2D planar surface corresponds to a single 3D planar surface in 3D space (Fig.3). Therefore,
in the same way as the linear consistency, by evaluating the degree of the smoothness of these 3D patches,
the correctness of the 2D-3D registration can be estimated. In this paper, we define this condition as theplanar consistency.
In order to evaluate the planar consistency, the covariance matrix of the 3D positions of the patches ofthe back-projected planar surface P Sk (0 ≤ k <N umps , N umps is the number of detected planar
surfaces.) and eigenvalues of the covariance matrix
are calculated. Here, three eigenvalues are defined
as µmin ( k ; R, t), µmed ( k ; R, t), µmax ( k ; R, t)(0 < µmin ( k ; R, t) < µmed ( k ; R, t) <µmax ( k ; R, t)), and eigenvectors are defined as
Ymin ( i ; R, t), Ymed ( i ; R, t), Ymax ( i ; R, t), respectively.
Ymax ( i ; R, t), Ymed ( i ; R, t), andYmin ( i ; R, t) indicate the directions corresponding
to the maximum, medium, and minimum variances, respectively. Therefore, the smaller the µmin ( k ; R, t)
is, the higher the possibility that back-projected planar
surface is aligned on a planar surface in the 3D modelis.
Thus, the planar consistency is defined as the following
equation.

Eplanarity (R, t) =µmin ( k ; R, t) (2)0≤k<N umps

Ymin ( i ; R, t) is the normal vector of the approximated 3D planar surface of the back-projected planar
surface P Sk in 3D space.
• Parallel and orthogonal consistencies
Parallel and orthogonal consistencies are defined forstraight lines and planar surfaces, respectively. In indoor environments, two planar surfaces are usually located in parallel or orthogonal, and thus 2D straightProjected planar surface
Planar surface

Projected planar surface
Planar surface

EplaneOrthogonality (R, t) = q   Ymin ( k ; R, t) · Ymin ( l ; R, t)  (6)
kl

View
Point

k>l
2D image

2D image
Geometric model(a)Here, pkl and qkl(0 ≤ pkl , qkl≤ 1) indicate the possibility that the relation between back-projected planar
surfaces P Sk and P Sl has parallel or orthogonal conditions. In case that pkl = 1 or qkl= 1, two backprojected planar surfaces have parallel or orthogonal
relations, respectively.

Geometric model(b)

Figure 3. Planar consistency.
lines and 2D planar surfaces have parallel or orthogonal relations with each other in 3D space, respectively.
In case that the relative position between image and
3D model is estimated precisely, two back-projected
2D straight lines and two back-projected 2D planar
surfaces on the 3D model have parallel or orthogonal
relations in most cases. On the other hand, in casethat the relative position between them is not estimatedcorrectly, neither parallelism nor orthogonality are satisfied for the back-projected 2D straight lines and 2D
planar surfaces on the 3D model (Figs.4 and 5). Therefore, by evaluating the degree of the parallelism or orthogonality of the 3D patches on which the 2D straightlines and 2D planar surfaces are back-projected, thecorrectness of the 2D-3D registration can be estimated.
In this paper, we define these consistencies as the parallel and orthogonal consistencies of straight lines andplanar surfaces.
Firstly, the parallel and orthogonal consistencies ofstraight lines are defined by calculating the eigenvector Xmax ( i ; R, t) and Xmax ( j ; R, t) of the backprojected straight lines SLi and SLj (i > j) as thefollowing equations.
ElineP arallelity (R, t) = pij Xmax ( i ; R, t) × Xmax ( j ; R, t)(3)i>jElineOrthogonality (R, t) = qij  Xmax ( i ; R, t) · Xmax ( j ; R, t)  (4)i>jHere, pij and qij (0 ≤ pij , qij ≤ 1) indicate the possibility that the relation between back-projected straightlines SLi and SLj has parallel or orthogonal conditions. In case that pij = 1 or qij = 1, two backprojected straight lines have parallel or orthogonal relations, respectively.
Next, the parallel and orthogonal consistencies of planar surfaces are defined by calculating the eigenvector Ymin ( k ; R, t) and Ymin ( l ; R, t) of the backprojected planar surfaces P Sk and P Sl (k > l) as thefollowing equations.
EplaneP arallelity (R, t) = pkl Ymin ( k ; R, t) × Ymin ( l ; R, t)(5)k>lStraight lines

Projected straight lines

Straight lines

Projected straight lines

View
Point

2D image

2D image

Geometric model

Geometric model(a) Parallel consistency(b) Orthogonal consistency

Figure 4. Parallel and orthogonal consistencies of straight lines.
Projected planar surfaces
Projected planar surfaces
Planar surfaces
Planar surfaces
View
Point
2D image

2D image
Geometric model(a) Parallel consistency

Geometric model(b) Orthogonal consistency

Figure 5. Parallel and orthogonal consistencies of planar surfaces.
3.1.3

Estimation of relative position and parallel andorthogonal conditions.
The relative position of the 2D image and the 3D modelis estimated by minimizing an evaluated value E, which isdefined as a linear sum of the evaluated values for the linear, planar, parallel and orthogonal consistencies shown inEqs.(1) ∼ (6).
E(R, t) = Elinearity (R, t) + α1 ElineP arallelity (R, t)+α2 ElineOrthogonality (R, t) + α3 Eplanarity (R, t)+α4 EplaneP arallelity (R, t) + α5 EplaneOrthogonality (R, t)(7)
In addition to the above procedure, it is possible to estimate the parallel and orthogonal conditions at the sametime by the following procedure. Firstly, P = {pij }i>j ,
Q = {qij }i>j , P = {pkl }k>l , and Q = {qkl}k>l are regarded as unknowns, and Eqs.(3) ∼ (6) are redefined as thefollowing equations.
ElineP arallelity (R, t, P) = 
pij Xmax ( i ; R, t) × Xmax ( j ; R, t) + c1
(1.0 − pij )
i>j

i>j

ElineOrthogonality (R, t, Q) =






qij (Xmax ( i ; R, t) · Xmax ( j ; R, t)) + c2
(1.0 − qij )estimated relative position. The edge-based registration isbased on correspondences between 2D photometrical and
3D geometrical edges [6]. In this method, photometrical
EplaneP arallelity (R, t, P ) = and geometrical edges extracted from a texture image andpkl Ymin ( k ; R, t) × Ymin ( l ; R, t) + c3
(1.0 − pkl ) a 3D geometric model are registered and the relative pok>lk>lsition is estimated so that these edges coincide with each
EplaneOrthogonality (R, t, Q ) =other. In our implementation, we use straight lines defined

 

 

(Ymin ( k ; R, t) · Ymin ( l ; R, t)) + c4
qkl
(1.0 − qkl) in section 3.1 as photometrical edges and geometrical edges
are extracted by detecting intersection lines between planark>lk>l(8) surfaces.
First, geometrical edges are projected to the texture imwhere c1 ∼ c4 are constants. By adding the new term toage as shown in Fig.6, and nearest photometrical edges fromEqs.(3) ∼ (6) and minimizing Eq.8, pij , qij , pkl , and qkl
geometrical edges are searched by using the k-D tree strucconverge to 0 or 1 after iterative estimation. Therefore theture. Then, the evaluated value Ede is defined as the folevaluated value E is defined as follows:
lowing equation using the distance di [pixel] between theseE(R, t, P, Q, P , Q ) = Elinearity (R, t)edge points. We define this value as the distance error.
N umde+α1 ElineP arallelity (R, t, P) + α2 ElineOrthogonality (R, t, Q)
di
i
E
(R,
t)
=
(12)
de
+α3 Eplanarity (R, t) + α4 EplaneP arallelity (R, t, P )N umde+α5 EplaneOrthogonality (R, t, Q ) + α6 ElineP enalty (P, Q)
where N um is the number of the projected geometrical
i>j

i>j

de

+α7 EplaneP enalty (P , Q )(9) edge points to the texture image.
To update parameters of the relative position between thewhere α1 ∼ α7 are constants, and ElineP enalty and2D
image and the 3D model, a force fi is applied to the
EplaneP enalty are additional constraints that two straightgeometricaledge point P. The force fi is the vector whichlines or two planar surfaces have never parallel and orthogis on a perpendicular line from the geometrical edge point
onal relations with each other at the same time.
 TheseP to the stretched line between the optical center and theterms are defined as ElineP enalty(P,Q)=
i>j pij qijphotometricaledge point. Using the force fi , current pose 
and EplaneP enalty (P , Q ) =k>l pkl qkl , respectively.
Randpositiont are updated as follows:
At the beginning of the estimation process, parallel and orN
umdethogonal conditions are set to pij = qij = pkl = qkl= 0.5
R ← R+αfi × (P − G)as the initial values, and are estimated using intermediateiparameters mij , nij , mkl , nkl as follows:
N
umde
∂E
∂E
t ← t+βfi(13)nij,t = nij,t−1 −mij,t = mij,t−1 −
∂pij
∂qij
i
∂E
∂E
where α and β are constants, and G is the center of gravitymkl,t = mkl,t−1 − nkl,t = nkl,t−1 −  (10)
∂pkl
∂qkl
of the 3D model. These procedures are repeated until the
distance error Ede converges.
where t is the iteration number of convergent calculation,diand pij , qij , pij , and qij
are defined by the sigmoid function.
fi
pij =pkl =

H

1
−

mij

1 + e ST1
1+em
− S kl
T

qij =

qkl
=

1
−

1+e

1
−

1+e

Nearest photometricaledge point

y

nij
ST

Focul point
n
kl
ST

l
G

u

(11)

where ST is a constant of the sigmoid function. By using
the sigmoid function, the range of pij , qij , pkl and qkl
canbe limited from 0 to 1. In our implementation, we minimize
the evaluated value E in Eq.(9) by steepest descent method
or conjugate gradient method.

3.2. Precise registration based on edge correspondences
After the geometrical consistency-based registration isconverged, the edge-based registration is applied from theP

3D geometric model

Projected geometricaledge point
2D texture image

Figure 6. Distance error between the photometrical and geometrical edges.

4. Experiments
In this section, we show some experimental results of the
2D-3D registration using a simulated 3D model and actual
environments measured by range and image sensors. In thesimulation, we focus on the convergence performance ofEvaluated value E

the proposed method against local minimums and do notconsider the effect of noise on observations.

Distance error
54

14000
12000

48

Distance error

40

10000

4.1. Comparison of the accuracy of geometrical consistency-based and edgebased registration methods
Firstly, we examined the change of the evaluated values of the geometrical consistencies(Eq.(7)) and the distance error(Eq.(12)) for various relative positions between
image and 3D models. Here, the parallel and orthogonal conditions of straight lines and planar surfaces are assumed to be known. The simulated 3D model shown in
Fig.7 (a) is an indoor environment which is 5.0 meters
high, 15.0 meters width, and 10.0 meter depth, and Fig.7(b) shows the 2D image obtained by capturing the textured 3D model. In these experiments, the pose and position errors between the 2D image and the 3D model are
set to (θroll , θpitch , θyaw )[degree] and (tx , ty , tz )[m], where−30 ≤ θ ≤ 30 and −3 ≤ t ≤ 3.
Figure 8(a) shows the evaluated values and Fig.8(b)is a close up of Fig.8(a) around the correct position. The horizontal axis indicates the pose er222ror (± θroll+ θpitch+ θyaw) and position error(± t2x + t2y + t2z ), and the left and right vertical axes
indicate the evaluated value E of the geometrical consistencies and the distance error Ede , respectively. From
Fig.8(a), though the distance error Ede has many local
minimums, we can see the evaluated value E is decreasingalmost monotonically to the correct position. This meansthat in the case that an initial registration error is large,
the relative position between the 2D image and the 3Dmodel can be estimated more robustly by the geometricalconsistency-based registration method comparing with theedge-based registration method. On the other hand, as
shown in Fig.8(b), the distance error converges steeply
around the correct position, while the evaluated values
E converges gradually. This indicates the edge-based
registration method is more suitable for the precise alignment than the geometrical consistency-based registration
method. From these consideration, we can conclude thatthe proposed method which combines these two methods
has both properties of robustness for initial registration
error and high accuracy for precise registration.

(a) Simulated 3D model(b) 2D image

Figure 7. 3D geometrical model and its textured image.

8000

32

6000

24

4000

16

Evaluated value E

8

2000
0
-50

-40

-30

-20

-10

-5

-4

-3

-2

-1

0

10

0

1

(a)

20
2

30
3

40
4

Evaluated value E

050 Orientation error [degree]
5 Position error [m]Distance error
24

6000

20

5000

Distance error

4000

16
12

3000

8

2000

Evaluated value E

1000

0

4

-5

-4

-3

-2

-1

-0.5

-0.4

-0.3

-0.2

-0.1

0

1

2

3

4

0

0.1

0.2

0.3

0.4

0
5 Orientation error [degree]
0.5 Position error [m](b)Figure 8. Comparison of evaluated value of
Eqs.7 and 12.

4.2. Evaluation of robustness against initial
registration errors
4.2.1

2D-3D registration in case that parallel and orthogonal conditions are known.

Next, we conduct the 2D-3D registration experiments using
the simulated 3D model on the condition that the paralleland orthogonal relations of straight lines and planar surfaces
are known. The performance of the convergence of threemethods, that is, the geometrical consistency-based registration method, the edge-based registration method, and the
proposed method, is compared. In these experiments, theinitial position is changed 100 times in a random manner.
The maximum values of the initial registration errors are
2 meters and 20 degrees for each axis. Figure 9 shows
the distribution of the frequencies of position and orientation errors after convergent calculation by the geometrical consistency-based registration method, the edge-based
registration method, and the proposed method. Examplesof the experimental results with the edge-based registration
method and the proposed method are shown in Figs.10 and
11.
As seen in these results, the edge-based registration
method estimates the relative position between the 2D image and the 3D model precisely if the relative position isconverged to the correct position. However, in the casethat the initial position is far from the correct position, it
is likely to get stuck into local minimums. On the other
hand, though the accuracy of the geometrical consistencybased registration algorithm is lower than the edge-based
registration method, the possibility of local minimums of
the proposed method is lower than the edge-based registration method. In the proposed method, the geometricalbased method could estimate the relative position close to
the correct position firstly, and then the edge-based method

3D geometrical edge

was applied for the precise pose estimation from the initial position estimated by the geometrical-based method.
Hence the proposed method estimated the relative position
precisely, and the success rate in which the distance error
becomes under 1.5 [pixel] is 96.0 %.
60

Frequncies [times]

60

50

50

40

40

30

30

20

20

2D image Geometrical consistency-based registration

10

10
0

Frequncies [times]

0

5

10

15
20
(a1) Orientation error

25

Orientation error[degree]300Position error
[m]
0

0.5

1.0

1.5
2.0
(a2) Position error

2.5

3.0

(a) Geometrical consistency-based registration algorithm
60

Frequncies [times]

60

50

50

40

40

30

30

20

20

10
0

0

5

10

15
20
(b1) Orientation error

25

Orientation error[degree]30

Frequncies [times]

Edge-based registration

10
0

0

0.5

1.0

1.5
2.0
(b2) Position error

2.5

Position error[m]3.0(b) Edge-based registration algorithm
60

Frequncies [times]

60

Figure 11. 2D-3D registration with the proposed method.
60

Frequncies [times]

60

50

50

50

50

40

40

40

40

30

30

30

30

20

20

10
0

0

5

10

15
20
(c1) Orientation error

25

Orientation error[degree]30

100Frequncies [times]

Frequncies [times]

0

0.5

1.0

1.5
2.0
(c2) Position error

2.5

Position error[m]3.0(c) Proposed method

Figure 9. Distribution of the frequencies afterconvergence. 3D geometrical edge

2D image

Figure 10. 2D-3D registration with the edgebased registration method only.
4.2.2 2D-3D registration in case that parallel and orthogonal conditions are also estimated.
We conduct the experiments to estimate the parallel or orthogonal conditions of straight lines and planar surfaces automatically by minimizing the evaluated value E detailed inEq.(9).
Figure 12 shows the distribution of the frequencies of
position and orientation errors after convergent calculationby the proposed method. The success rate in which the distance error becomes under 1.5 [pixel] is 75.0 %. Figure 13
shows examples of the estimated results of the parallel andorthogonal conditions. These results show that the paralleland orthogonal conditions are estimated correctly. For all of
the straight lines, the success rates of the estimation of theparallel and orthogonal conditions are 85.7 % and 83.5 %,respectively. For all of the planar surfaces, the success rates
are 100.0 %.

20

20

10

10
Orientation error[degree]
0
0
30

0

0

5

10

15
20
Orientation error

25

0.5

1.0

1.5
2.0
Position error

2.5

Position error[m]
3.0

Figure 12. Distribution of the frequencies after convergence.

4.3. 2D-3D registration using actual models
Finally, we carry out an experiment of 2D-3D registration using an actual environment measured by a range
sensor (SICK, LMS200) and an image sensor (FUJIFILM,
FinePix). In this experiment, the condition of parallel andorthogonal relations are assumed to be unknown and estimated. Figures 14 (a) and 14 (b) show the actual 3D geometrical model of a room and its corresponding 2D image. Figure 14 (c) shows the experimental results of estimation of relative pose between the 2D image and the 3Dmodel. From these experiments, we can conclude that that
2D-3D registration is performed successfully for actual environments.

5. Conclusion
This paper proposed a new registration algorithm of a
2D image and a 3D geometrical model for creating realistic 3D model of indoor scene settings. In this method,
firstly the relative position between the 2D image and the
3D model is estimated utilizing geometrical consistencies
of back-projected 2D photometrical edges on the 3D model.
Next, the edge-based method is applied after the geometrical consistency-based registration method is converged forthe optimum pose estimation.
Our registration algorithm realizes the robust estimationof relative position between the 2D image and the 3D model
against initial registration errors. This method works welleven if the features extracted from a 2D image or a 3D

PS1
SL2

SL3

SL1

PS2

PS3

SL4
PS4
SL5
PS5

1.0

Straight lines

qij

1.0

Planar surfacespij(a) 3D model(b) 2D image

Geometrical edge
0.8

0.8
SL1 and SL5

0.6

1.0

SL1 and SL3
0.4

SL1 and SL2
SL1 and SL3

0.2

0

0.6

SL1 and SL4

0.4

0

200

400
600
800 1000(a1) Orthogonal condition

SL1 and SL2
SL1 and SL4

0.2
Iteration[times]1200

1400

SL1 and SL5
0

0

200

(a) Straight lines
p'kl

q'kl

Iteration[times]400
600
800 1000(a2) Parallel condition

1200

1400

1200

1400

Geometrical consistency-based registration

1.0

0.8

0.8

PS1 and PS5
PS1 and PS2
PS1 and PS4
PS1 and PS3

0.6

0.6

0.4

0.4

PS1 and PS5

0.2

0

0

200

400
600
800 1000(b1) Orthogonal condition

PS1 and PS2
PS1 and PS3
PS1 and PS4

0.2
Iteration[times]
1200

1400

0

Edge-based registration
Iteration[times]

0

200

400
600
800 1000(b2) Parallel condition(b) Planar surfaces

Figure 13. Estimation of parallel and orthogonal conditions of straight lines and planar
surfaces.
model are limited such as for indoor environments. The
relevance of the proposed method was verified through theexperiments using simulated 3D models of indoor scene settings and actual environments measured by image and range
sensors.

References[1] L. Brunie, S. Lavallee, and R. Szeliski. Using force fields
derived from 3d distance maps for inferring the attitude of a
3d rigid object. Proc. of the Second European Conference
on Computer Vision, pages 670–675, 1992.
[2] Q. Delamarre and O. Faugeras. 3d articulated models andmulti-view tracking with silhouettes. Proc. of the Int. Conf.
Computer Vision, 2:716–721, 1999.
[3] P. E. Devebec, C. J. Taylor, and J. Malik. Modeling andrendering architecture from photographs:a hybrid geometryand image-based approach. Proc. of ACM SIGGRAPH’96,
pages 11–20, 1996.
[4] M. D. Elstrom and P. W. Smith. Stereo-based registration ofmulti-sensor imagery for enhanced visualization of remoteenvironments. Proc. of the 1999 IEEE Int. Conf. Robotics
and Automation, pages 1948–1953, 1999.
[5] Y. Iwashita, R. Kurazume, K. Hara, and T. Hasegawa. Fastalignment of 3d geometrical models and 2d color images
using 2d distance maps. Proc. of The 5th Int. Conf. 3-D
Digital Imaging and Modeling, pages 164–171, 2005.
[6] R. Kurazume, K. Nishino, Z. Zhang, and K. Ikeuchi. Simultaneous 2d images and 3d geometric model registration for
texture mapping utilizing reflectance attribute. Proc. of Fifth
Asian Conference on Computer Vision, pages 99–106, 2002.
[7] S. Lavallee and R. Szeliski. Recovering the position andorientation of free-form objects from image contours using

Mapping the camera image
on the 3D model(c) Registration process

Figure 14. 2D-3D registration using actualmodels.
3d distance maps. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 17(4):378–390, 1995.
[8] H. Lensch, W. Heidrich, and H. P. Seide. Hardwareaccelerated silhouette matching. In SIGGRAPH Sketches,
2000.
[9] H. Lensch, W. Heidrich, and H.-P. Seidel. Automated texture registration and stitching for real world models. In Pacific Graphics ’00, pages 317–326, 2000.
[10] H. Lensch, W. Heidrich, and H.-P. Seidel. A silhouettebased algorithm for texture registration and stitching.
Graphical Models, 63:245–262, 2001.
[11] H. Lipson and M. Shpitalni. Optimization-based reconstruction of a 3d object from a single freehand line drawing. J.
Computer Aided Design, 28(8):651–663, 1996.
[12] L. Liu and I. Stamos. Automatic 3d to 2d registration for thephotorealistic rendering of urban scenes. Proc. of Computer
Vision and Pattern Recognition, 2:137–143, 2005.
[13] K. Matsushita and T. Kaneko. Efficient and handy texture
mapping on 3d surfaces. Comput. Graphics Forum 18, pages
349–358, 1999.
[14] P. Neugebauer and K. Klein. Texturing 3d models of
real world objects from multiple unregistered photographic
views. Comput. Graphics Forum 18, pages 245–256, 1999.
[15] I. Stamos and P. K. Allen. Automatic registration of 2-d with
3-d imagery in urban environments. Proc. of the Int. Conf.
Computer Vision, pages 731–737, 2001.
[16] K. Umeda, G. Godin, and M. Rioux. Registration of range
and color images using gradient constraints and range intensity images. Proc. of 17th Int. Conf. Pattern Recognition,
pages 12–15, 2004.
[17] S. Zuffi, A. Leardini, F. Catani, S. Fantozzi, and A. Cappello.
A model-based method for the reconstruction of total knee
replacement kinematics. IEEE Trans. on Medical Imaging,
18(10):981–991, 1999.