ACCV2002: The 5th Asian Conference on Computer Vision, 23–25 January 2002, Melbourne, Australia

Simultaneous 2D images and 3D geometric model registration for texture
mapping utilizing reflectance attribute
Ryo Kurazume

Ko Nishino

Zhengyou Zhang

Katsushi Ikeuchi

The University of Tokyo The University of Tokyo Microsoft Research, Inc. The University of Tokyokurazume,kon,ki@cvl.iis.u-tokyo.ac.jp

Abstract

3. how to integrate such virtual objects with real or otherVR scenes

Texture mapping on scanned objects, that is, the method tomap current color images on a 3D geometric model measured by a range sensor, is a key technique of photometricmodeling for virtual reality. Usually range and color images are obtained from different viewing positions, throughtwo independent range and color sensors. Thus, in order tomap those color images on the geometric model, it is necessary to determine relative relations between these two viewpoints. In this paper, we propose a new calibration method
for the texture mapping; the method utilizes reflectance images and iterative pose estimation based on a robust Mestimator.
Moreover, since a 2D texture image taken from one viewing point is a partial view of an object, several images must
be mapped onto the object in order to cover the entire 3D
geometric model. In this paper, we propose the new simultaneous registration technique of several images and geometric model based on 2D-3D edge correspondence and the
epipolar constraint between images.

For geometric modeling, we have developed a three-stepmethod: mesh generation, alignment, and merging [7]. For
photometric modeling, we have developed two rendering
methods, model-based [1] and eigen-texture rendering [2],both of which automatically create such rendering models
by observing real objects. For integration of virtual object
with real scenes, we have developed a method that renders
virtual objects based on real illumination distribution [3].
Texture mapping on scanned objects, that is, the method tomap current color images on a 3D geometric model measured by a range sensor, is also a key technique of photometric modeling for virtual reality [5],[9],[10]. With the exception of several range sensors which have special configurations of optics, such as OGIS or Cyberwares, and whichtake both range and color images from the same viewing
position, range and color images are usually obtained fromdifferent viewing positions, through two independent range
and color sensors. In order to map these color images on
the geometric model, it is necessary to determine the relative relationship between these two viewpoints.
In this paper, we propose a new calibration method for
texture mapping using reflectance images, which are given
as side products of range images for most of the range sensors. In this method, 3D reflectance edge points and 2Dcolor edge points are first extracted from the reflectance andthe color images, respectively. Then, the relative pose ofeach sensor is determined so that 3D reflectance edge points
are aligned with the 3D points calculated from 2D color
edge points by using the iterative pose estimation based on
the robust M-estimator [7], [8].
Moreover, since a 2D texture image taken from one
viewing point is a partial view of an object, several images
must be mapped onto the object in order to cover the entire 3D geometric model. In this paper, we propose the new
simultaneous registration technique of several images andgeometric model based on 2D-3D edge correspondence and

1 Introduction
One of the most important research issues in virtual reality (VR) is how to obtain models for virtual reality. Currently, such models are manually created by a human programmer. This manual method requires long development
time and the resulting virtual reality systems are expensive.
In order to overcome this problem, we have been developing techniques to automatically create virtual reality models
through observation of real objects; we refer to these techniques as modeling-from-reality (MFR). The MFR methodspans three aspects:
1. how to create geometric models of virtual objects
2. how to create photometric models of virtual objects
1geometric constraint between images, that is so called the
epipolar constraint.

once the relationship is calibrated. Such a fixed structure isinconvenient. Usually, a color camera is much handier thana range sensor. We prefer to take color images freely without having to transport a heavy range sensor.
Generally speaking, range sensors often provide reflectance images as side products of range images. A reflectance image is given as a collection of the strength ofreturned laser energy at each pixel. This reflectance imageis aligned with the range image because both images are
obtained through the same receiving optical device. The returned timing provides a depth measurement, while the returned strength provides a reflectance measurement. Thus,
this reflectance image is commonly available in a wide variety of range sensors, including ERIM, Perceptron, and ourmain sensor, Cyrax.
We decided to employ this reflectance image as a vehicle
for the alignment of range images with intensity images.
Reflectance images are similar to intensity images in thatboth images are somehow related by surface roughness. As
mentioned above, the reflectance image is aligned with therange image. Thus, to align the reflectance image with theintensity image is a much easier task than that of aligning
the range image with the intensity image.
More precisely, we align edges extracted from reflectance images by the Canny operator with those in intensity images so that 3D position error of those edges isminimized by the iterative calculation. Edges in reflectanceimages are generated due to several reasons. A boundary
between two different colors or materials generates a discontinuity of reflectance and, thus, an edge in a reflectanceimage. For instance, since our Cyrax range scanner uses agreen laser diode, reflectance edges can be observed along
the boundary between two colors or materials which havethe different reflectance ratios for this wavelength. Since
different materials are of different colors, a discontinuity
also appears in the color images. In particular, an intensity image with the same wavelength as that of a laser scanner, the G channel for our Cyrax range scanner, has robust
similarities with the reflectance images. Jump edges along
small ranges in a range image also appear as jump edges inreflectance images. Such jump edges often provide small
shadow regions alongside themselves and are also observed
as edges in intensity edges. Occluding boundaries are observed in reflectance and intensity images. By aligning thediscontinuities in reflectance images with those in intensity
images, we can determine the relative relationship between
the range and intensity views.
Prior to doing the alignments, we paste the necessaryreflectance edges onto the 3D geometric model. As mentioned above, since the occluding boundaries vary depending on the viewing direction, the edges along the occluding boundaries are previously removed from the reflectanceimages. However, the edges along the current occluding

2 Related work
For alignment of range and color images, Viola proposed atechnique based on a statistical method [6]. Allen et al. also
proposed a method using the intersection lines of several
planes extracted from the range data and the color edges [5].
Several object recognition algorithms also can be used toalign range edges with intensity edges[7],[8]. These methods work well on the surface with little albedo variance. But
these are highly likely to be trapped to a local minimum on
surfaces with rich textures.
Elstrom et al. proposed an alignment method using reflectance images [4]. This method can be summarized asfollows:
1. Extract feature points from the reflectance and intensity images by the use of a corner detector.
2. Determine correspondence between these featurepoints by calculating the similarity based on gray level
subtraction and shape of corners.
3. Estimate the relative pose roughly by solving theclosed form equation derived from the determined correspondence.
4. Estimate the relative pose precisely by comparing thedepth of feature points obtained by a stereo calculation
using reflectance and intensity images, with the featurepoint depth obtained by a laser measurement.
This method works well in cases where the object consistsof flat planes with few texture patterns and only simple line
edges are extracted. However, the corner detector has low
reliability for the object that consists of curved surfaces. In
addition, it is difficult to determine good correspondence offeature points between the reflectance and intensity images,
which are obtained from different configurations of optics.
Furthermore, since contour lines of curved surfaces depend
on a viewing direction, the contour edges of the reflectanceimage do not directly correspond with the contour edges of
the color image. Thus, for accurate pose estimation, thesecontour edges should be removed in advance from the extracted reflectance edges.

3 Aligning a texture on range data
One simple solution for determining the relationship between range and color images is through calibration using
the calibration board and fixtures. However, this method requires that the range and color sensors be fixed on the fixture2

boundaries can be estimated from the 3D geometric modeland the current viewing direction. Our algorithm extracts
them automatically, and uses them for the alignment. Thus,
this alignment problem is the one between:

3D geometric model. The edges along the occluding boundary will also be sampled into edge points along the occluding boundary, which is detected as  3D reflectance edges existing on 3D range surfaces
 3D edges along the occluding boundaries existing on
3D range surfaces               (2)where is a threshold. Pruning of small edges is necessary.
Currently, by adjusting various parameters in edge tracking,we remove small edges and extract only dominant edges.

and 2D edges in the 2D intensity image

3.2

Correspondences

To establish correspondence, the system projects 3D edge
points onto the 2D image plane. Then, the system finds thenearest 2D edge points on the image plane. This point pair
will be used to evaluate the value of the error function in
next step.

Extracted edges are represented as a collection of points
along them as shown in Fig.1. Consequently, the alignmentis done between 3D points on 3D range surfaces and 2D
points in the 2D image, that is, as the alignment between
3D and 2D points.

3.3

Edge linePose estimation by M-estimator

To determine the relative pose that coincides the position of
2D intensity edges and 3D edges, we use the M-estimator,
which is one of the robust estimators.
First, the distance between corresponding 2D intensity
edge points and 3D edge points is evaluated as shown in
Fig.2: where  is a 3D error vector which is on a perpendicular line from a 3D edge point to the stretched line between the optical center and a 2D edge point on the image
plane.
    (3)

Sample points

Figure 1: Edge sampling
The three steps in the 3D-2D alignment are as follows:
1. Extract possible observable reflectance edges on 3D
reflectance images and edges along the occluding
boundaries based on the current viewing direction.

where  is the distance between the optical center and 3Dedge point, and  is the angle between 2D intensity edge
point and 3D edge point.
The system finds the configuration,  , which minimizes
the total error,  ,2. Establish correspondences between 3D edges and 2D
intensity edges.
3. Determine the relative relationship iteratively based on
the correspondences.

    

(4)



3.1

where  is an error function. The minimum of    can beobtained by:

Visibility calculation and correspondences     


Applying Canny operators to the reflectance images enables
us to obtain reflectance edges. These are then sampled toedge sampling points, and stored on the surface of the 3D
geometric model. First, the system decides which edge partis visible from the current viewing direction. Visibility ofan edge point,  , can be examined as

 



             

We can consider
terms.

(5)

 as a weight function to evaluate error   
(6)
 

Thus, the minimization can be derived as the following leastsquared equation:

(1)


 
  





where  is the surface normal of the patch and  is the current camera’s viewing direction with respect to the current
3(7)

We choose the Lorentzian function for this function.

       

3D edge
2D edge(8)

By solving this equation using the conjugate gradientmethod, we can obtain the configuration  that minimizes
the error term and gives the relative relationship between
the camera and range sensor.

H(2) After 1 iteration(3) After 2 iterations(4) After 5 iterations

Figure 4: Simulation resultszilP (x,y,z)

Table 1: Position errors [mm (pixel)]

2D Color Edge

d

θ

y

3D Edgeu

Projection of 3D Edge

Average
STD.

Zi

Figure 2: 2D distance and 3D distance

3.4(1) Initial positionx
0.14 (0.12)
0.13 (0.11)

y-0.20 (0.16)
1.89 (1.56)

z-967.81
5.94

[deg.]
4.0
4.1

In Table 1, the depth of the z axis is calculated closer
than the actual position. This is because 3D edge points
along the occluding boundaries are obtained as the center oftriangle patches which satisfy Eq.2, and these center points
are calculated inside the actual boundaries. Thus, if the triangle patches were to be set more densely, this would not
be a problem.

Experiment3.4.1 Computer simulation
In order to verify the performance of the proposed 2D-3Dalignment method, the computer simulation was carried out.
In this simulation, we assumed that a cylinder with 100 mm
diameter and 200 mm length was placed 1000 mm aheadof the color and range sensors. We did not assume the reflectance values obtained from the range sensor, and used
3D edge points along the occluding boundaries only. The
resolution of the color sensor was 300 dpi and focus length
was 70 mm.
Fig.3 shows the simulation model, and Fig.4 shows theprocess of the alignment.

As a result of this simulation, it wass verified that theaccurate relative pose between the camera and the range
sensor can be estimated by the proposed 3D-2D alignment
method.

3.4.2 Texture mapping of a dish with drawing
Next, using the Cyrax laser scanner , we attempted the texture mapping of a dish with drawing.
Fig.5 shows the reflectance image and extracted reflectance edges, and Fig.6 shows the color texture image
taken by the digital camera (Nikon, D1) along with the extracted intensity edges. Here, as shown in Fig.5, the brighter
the pixel color is, the larger its reflectance.

yx

Fig.7 shows the procedure of the alignment of the reflectance and intensity images through the use of the Mestimation method. For these symmetrical objects, accuratealignment which coincides with the position of the drawing
is impossible for the conventional methods which use thegeometric edges extracted from the geometric model or the
CAD model. On the other hand, our proposed method canperform accurate alignment due to the use of reflectance information obtained from the drawing as shown in Fig.8

z

Figure 3: Simulation model
From several initial poses with maximum 50 mm and
20 degrees difference from the actual pose, the optimum
pose was calculated. Table 1 shows the obtained averageand the standard deviation of the estimated position and theinclination of the y axis after 10 simulations.
4Figure 8: Aligned color texture on the dish(a) Reflectance image(b) Reflectance edges

Figure 5: Reflectance image of the dish(a) Color image(b) Intensity edges

Figure 9: Geometric model of the Great Buddha of Kamakura

Figure 6: Texture image of the dishcedure for the alignment of the reflectance and intensity images through the use of the M-estimation method.
By comparing Fig.10 and Fig.11, the reflectance andthe color image have robust similarities, for example, theshapes of rust on the forehead or the junctures of bronze.
Fig.13 shows the result of the mapping of the current color
texture onto the 3D geometric model.

3.4.3 Texture mapping of Great Buddha of Kamakura
We have recently developed a method for preserving
Japanese cultural heritage objects. Using accurate laser
scanners [11], 3D digital models of these objects are created through observation. For our first preservation project,
we have created a 3D geometric model of the Great Buddha
of Kamakura as shown in Fig.9.
Using our proposed alignment method, we tried to map
the color texture image taken from the digital camera onto
this model. Fig.10 shows the reflectance image and extracted reflectance edges, Fig.11 shows the color texture
image taken by the digital camera. Fig.12 shows the pro-

Intensity edges

Figure 10: Reflectance image of the Great Buddha of Kamakura
Reflectance edges(1) Initial position(2) After 10 iterations

4(3) After 20 iterations

Simultaneous registration using geometric constraint

The 2D-3D registration technique proposed above is forthe alignment between one texture image and a geometricmodel. However, since a 2D texture image taken from one
viewing point is a partial view of an object, several images
must be mapped onto the object in order to cover the en-(4) After 30 iterations

Figure 7: Intensity edges aligned with reflectance edges
5Figure 11: Texture image of the Great Buddha of Kamakura
Figure 13: Aligned color texture on the 3D geometric model

Σw
Σ1

Mw

m1

m2
Σ2

Figure 14: Coordinate systems
Using the pseudo inverse
Eq.(10) is changed asÝ of the projection matrix  , Ý     Ý Ý   


(11)



Where,  is the position of the focus of the camera 2
expressed in the world coordinate system. By substituting
Eq.(11) into Eq.(9), we can derive

Figure 12: Aligned intensity edges with reflectance edges   Ý    Ý  
Ý    

tire 3D geometric model. One simple method to align theseveral images and the geometric model is to utilise the proposed 2D-3D registration technique for each image sequentially. However, as the additional criterion, we can utilize
the geometric constraint between these texture images, thatis so called the epipolar constraint [12].
Let’s consider the case that the two texture images, 1 and
2, which are taken from the two different viewing points,
are going to be mapped. First, we define the world coordinate system  and two camera coordinate systems thatis fixed on each viewing point,  and  , as shown in
Fig.14. Point          , which is an arbitrary point expressed in the world coordinate system, is projected on each image plane as points       and     according to the following equations.

Where,  is called as the “epipole”, which is the crossing
point of the line connecting the focus points of both cameras, and the image plane of camera 1. Beside, by multiplying  and  for both sides of the equation, thefollowing equation can be derived.

(9)

The left side of this equation is always 0. Consequently, we
can obtain the next equation.
















       Ý    Ý 






    (12)          Ý     
(10)

¢

are the projection matrixes
Where,  and      Ý        consisting of the internal and external camera parameters.

6

(13)
(14)

(15)
(16)

Where,is called as the fundamental matrix [12].
(17)both images must be determined beforehand. We can consider two kinds of corresponding point detection methods
as follows:

On the image plane of the camera 1, radial lines from theepipole can be written as Corresponding regions based on the similarity calculation of both images.

   Ý         Selected edge points on each image for the calculationof 2D-3D alignment proposed in Section 3.

(18)

These lines are called as epipolar lines. Therefore, if thecorresponding points on both images,  and , are determined, the distance between the point  and the epipolar line corresponding to the point  on image plane 1 canbe calculated as






 







  In the following experiments, we adopted the method basedon the similarity calculation.
Figure 15 shows the two texture images of the dish andFigures 16 and 17 show the procedure of the simultaneousand the separate registration of two texture images and a geometric model of a dish, respectively. The epipolar lines for
only three corresponding points are also shown in Figures
16. 12 corresponding points, depicted as the small rectangles in Figure 15, were calculated using KLT method [13].
In Figure 17, the registration processes were executed separately for each image without the use of epipolar constraint,
and it is clear that the registration of iamge 2 was failed dueto the local minimum. Figure 18 shows the mapping resultsfor each texture image and both images onto the geometricmodel of the dish.

          (19)

In the same manner, the distance of the epipolar line and thecorresponding point on the image plane 2 is given as









 


    

(20)

Therefore, the total distance of these corresponding points
is obtained as       



     












 

Image 1







where,



 








(21)
½
¾

(22)

 Equation (21) shows the geometric constraint of the focus
position of the images 1 and 2, and is called as the “epipolar
constraint”.
Therefore, simultaneous registration of two 2D texture
images and the 3D geometric model is formulated as theminimization problem of total error defined by next equation.

  
½ ¾







   









  

Figure 15: Two color images of the dish5 

Conclusions

In this paper, we proposed a new calibration method for texture mapping through the use of reflectance images, which
are given as side products of range images for most range
sensors. In this method, 3D reflectance edge points and 2Dcolor edge points are first extracted from the reflectance image and the color image, respectively. Then, the relativepose of each sensor is determined so that the 3D reflectanceedge points coincide with the 3D points calculated from the2D color edge points be the use of the robust M-estimator.
Furthermore, we propose the new simultaneous registration
technique of several texture images and geometric model

(23)



where,      and     are theerrors of the 2D-3D registration defined in Eq.(4),      is the error derived from the epipolar
constraint of both images as Eq.(21), and  is the error function.
To calculate the optimum poses  and  that minimize the error defined by Eq.(23), corresponding points of Image 2



7

Image 1

Image 2(a) Image 1(a) Initial position(b) Image 2(b) After 5 iterations(c) Mapping both images(c) After 40 iterations

Figure 18: Mapped texture images
GRAPH 97, In Computer Graphics Proceedings, Annual Conference Series 1997, ACM SIGGRAPH, pp.379-387, August 1997.
[2] K.Nishino, Y.Sato and K.Ikeuchi, ”Eigen-Texture Method: Appearance Compression based on 3D Model”, in Proc. of Computer Vision and Pattern Recognition ’99, vol.1, pp.618-624, Jun., 1999.
[3] I. Sato, Y. Sato, and K. Ikeuchi, ”Acquiring a radiance distribution
to superimpose virtual objects onto a real scene,” IEEE Trans Visualization and Computer Graphics, Vol. 5, No. 1, pp.1-12, January
1999.
[4] Mark D. Elstrom and Philip W. Smith,Stereo-Based Registration ofMulti-Sensor Imagery for Enhanced Visualization of Remote Environments,Proceedings of the 1999 IEEE International Conferenceon Robotics & Automation, pp.1948-1953, 1999.
[5] Ioannis Stamos and Peter K. Allen, Integration of Range and Image Sensing for Photorealistic 3D Modeling, Proceedings of the2000 IEEE International Conference on Robotics and Automation,
pp.1435-1440, 2000.
[6] P. Viola and W.M. Wells III, Alignment by maximization of mutual information, International Journal of Computer Vision, Vol.24,
No.2, pp.137-154, 1997.
[7] M. D. Wheeler, ”Automatic Modeling and Localization for Object
Recognition”, Technical Report (Ph.D. Thesis), CMU-CS-96-188,
School of Computer Science, Carnegie Mellon University, October,
1996.
[8] M. D. Wheeler and Katsushi Ikeuchi, ”Sensor Modeling, Probabilistic Hypothesis Generation, and Robust Localization for Object
Recognition”, IEEE Transactions on Pattern Analysis and MachineIntelligence ,Vol. 17, No. 3, March 1995.
[9] P. Debevec, D.J. Taylor, and J. Malik, Modeling and rendering architecture from photographs; A hybrid geometry and image-baseapproach, Proc. of SIGGRAPH’96, pp.11-20, 1996.
[10] P. Debevec, Y. Yu, and G. Borshukov, Efficient view-dependentimage-based rendering with projective texture-mapping 9th Eurographics workshop on rendering, pp.105-116, 1998.
[11] Daisuke Miyazaki, Takeshi Ooishi, Taku Nishikawa, Ryusuke
Sagawa, Ko Nishino, Takashi Tomomatsu, Yutaka Takase, Katsushi
Ikeuchi,The Great Buddha Project: Modelling Cultural Heritage
through Observation,VSMM2000 (6th international conference on
virtual systems and multimedia), pp.138-145, 2000.
[12] Z.Zhang, R. Deriche, O. Faugeras and Q,T, Kuong, A robust technique for matching two uncalibrated images through the recoveryof the unknown epipolar geometry. Artificial Intelligence Journal,
78: 87-119, 1995.
[13] Jianbo Shi and Carlo Tomasi, Good Features to Track, IEEE Conference on Computer Vision and Pattern Recognition, pp.593-600,
1994.

Figure 16: Simultaneous registration using epipolar constraint
Image 1

Image 2(a) Initial position(b) After 5 iterations(c) After 40 iterations

Figure 17: Separate registration without epipolar constraint
based on 2D-3D edge correspondence and the epipolar constraint between images.
Using the proposed method, experiments of the texture mapping for a symmetrical dish and the 3D geometricmodel of Great Buddha of Kamakura were carried out, andthe usefulness of the proposed method was verified.

Acknowledgments
This project is funded by Core Research for Evolutional
Science and Technology (CREST) of Japan Science andTechnology Corporation (JST).

References[1] Y. Sato, M. D. Wheeler, and K. Ikeuchi, ”Object shape and reflectance modeling from observation”, Proceedings of ACM SIG-

8