Mapping textures on 3D geometric modelusing reflectance image
Ryo Kurazume
The University of Tokyo

M. D. Wheeler
Katsushi Ikeuchi
Cyra Technologies, Inc.
The University of Tokyokurazume,ki@cvl.iis.u-tokyo.ac.jp

Abstract
Texture mapping on scanned objects, that is, the method tomap current color images on a 3D geometric model measured by a range sensor, is a key technique of photometricmodeling for virtual reality. Usually range and color images are obtained from different viewing positions, throughtwo independent range and color sensors. Thus, in order to map those color images on the geometric model, it
is necessary to determine relative relations between thesetwo viewpoints. In this paper, we propose a new calibration method for the texture mapping; the method utilizes
reflectance images and iterative pose estimation based ona robust M-estimator.

1

Introduction

One of the most important research issues in virtual reality(VR) is how to obtain models for virtual reality. Currently,
such models are manually created by a human programmer. This manual method requires long development time
and the resulting virtual reality systems are expensive. In
order to overcome this problem, we have been developing
techniques to automatically create virtual reality models
through observation of real objects; we refer to these techniques as modeling-from-reality (MFR). The MFR methodspans three aspects:
1. how to create geometric models of virtual objects
2. how to create photometric models of virtual objects
3. how to integrate such virtual objects with real or otherVR scenes
For geometric modeling, we have developed a three-stepmethod: mesh generation, alignment, and merging [1]. For
photometric modeling, we have developed two rendering
methods, model-based [2] and eigen-texture rendering [3],both of which automatically create such rendering models
by observing real objects. For integration of virtual object

with real scenes, we have developed a method that renders
virtual objects based on real illumination distribution [4].
Texture mapping on scanned objects, that is, the method tomap current color images on a 3D geometric model measured by a range sensor, is also a key technique of photometric modeling for virtual reality. With the exception ofseveral range sensors which have special configurations of
optics, such as OGIS or Cyberwares, and which take both
range and color images from the same viewing position,
range and color images are usually obtained from different viewing positions, through two independent range andcolor sensors. In order to map these color images on thegeometric model, it is necessary to determine the relative
relationship between these two viewpoints.
In this paper, we propose a new calibration method for texture mapping using reflectance images, which are given asside products of range images for most of the range sensors. In this method, 3D reflectance edge points and 2Dcolor edge points are first extracted from the reflectanceand the color images, respectively. Then, the relative poseof each sensor is determined so that 3D reflectance edge
points are aligned with the 3D points calculated from 2Dcolor edge points by using the iterative pose estimation
based on the robust M-estimator [1], [8].

2 Related work
For alignment of range and color images, Viola proposeda technique based on a statistical method [7]. Allen etal. also proposed a method using the intersection lines ofseveral planes extracted from the range data and the color
edges [6]. Several object recognition algorithms also canbe used to align range edges with intensity edges. Thesemethods work well on the surface with little albedo variance. But these are highly likely to be trapped to a local
minimum on surfaces with rich textures.
Elstrom et al. proposed an alignment method using reflectance images [5]. This method can be summarized asfollows:

1. Extract feature points from the reflectance and intensity images by the use of a corner detector.
2. Determine correspondence between these featurepoints by calculating the similarity based on gray level
subtraction and shape of corners.
3. Estimate the relative pose roughly by solving theclosed form equation derived from the determinedcorrespondence.
4. Estimate the relative pose precisely by comparing thedepth of feature points obtained by a stereo calculation using reflectance and intensity images, with thefeature point depth obtained by a laser measurement.
This method works well in cases where the object consistsof flat planes with few texture patterns and only simple line
edges are extracted. However, the corner detector has low
reliability for the object that consists of curved surfaces. In
addition, it is difficult to determine good correspondenceof feature points between the reflectance and intensity images, which are obtained from different configurations of
optics. Furthermore, since contour lines of curved surfaces
depend on a viewing direction, the contour edges of the reflectance image do not directly correspond with the contour
edges of the color image. Thus, for accurate pose estimation, these contour edges should be removed in advancefrom the extracted reflectance edges.

3

Aligning textures on range data

One simple solution for determining the relationship between range and color images is through calibration using
the calibration board and fixtures. However, this method
requires that the range and color sensors be fixed on the
fixture once the relationship is calibrated. Such a fixed
structure is inconvenient. Usually, a color camera is muchhandier than a range sensor. We prefer to take color images
freely without having to transport a heavy range sensor.
Generally speaking, range sensors often provide reflectance images as side products of range images. A reflectance image is given as a collection of the strength ofreturned laser energy at each pixel. This reflectance imageis aligned with the range image because both images are
obtained through the same receiving optical device. The returned timing provides a depth measurement, while the returned strength provides a reflectance measurement. Thus,
this reflectance image is commonly available in a wide variety of range sensors, including ERIM, Perceptron, andour main sensor, Cyrax.
We decided to employ this reflectance image as a vehicle
for the alignment of range images with intensity images.

Reflectance images are similar to intensity images in thatboth images are somehow related by surface roughness. As
mentioned above, the reflectance image is aligned with therange image. Thus, to align the reflectance image with theintensity image is a much easier task than that of aligning
the range image with the intensity image.
More precisely, we align edges extracted from reflectanceimages by the Canny operator with those in intensity images so that 3D position error of those edges is minimizedby the iterative calculation. Edges in reflectance images
are generated due to several reasons. A boundary between
two different colors or materials generates a discontinuityof reflectance and, thus, an edge in a reflectance image.
For instance, since our Cyrax range scanner uses a green
laser diode, reflectance edges can be observed along theboundary between two colors or materials which have thedifferent reflectance ratios for this wavelength. Since different materials are of different colors, a discontinuity also
appears in the color images. In particular, an intensity image with the same wavelength as that of a laser scanner, theG channel for our Cyrax range scanner, has robust similarities with the reflectance images. Jump edges along small
ranges in a range image also appear as jump edges in reflectance images. Such jump edges often provide small
shadow regions alongside themselves and are also observed as edges in intensity edges. Occluding boundaries
are observed in reflectance and intensity images. By aligning the discontinuities in reflectance images with those inintensity images, we can determine the relative relationshipbetween the range and intensity views.
Prior to doing the alignments, we paste the necessary reflectance edges onto the 3D geometric model. As mentioned above, since the occluding boundaries vary depending on the viewing direction, the edges along the occluding
boundaries are previously removed from the reflectanceimages. However, the edges along the current occluding
boundaries can be estimated from the 3D geometric modeland the current viewing direction. Our algorithm extracts
them automatically, and uses them for the alignment. Thus,
this alignment problem is the one between:

 3D reflectance edges existing on 3D range surfaces
 3D edges along the occluding boundaries existing on
3D range surfacesand 2D edges in the 2D intensity image
Extracted edges are represented as a collection of points
along them as shown in Fig.1. Consequently, the alignmentis done between 3D points on 3D range surfaces and 2D

points in the 2D image, that is, as the alignment between
3D and 2D points.

3.2

To establish correspondence, the system projects 3D edge
points onto the 2D image plane. Then, the system finds thenearest 2D edge points on the image plane. This point pair
will be used to evaluate the value of the error function in
next step.

3.3
Edge line

Sample points

Figure 1: Edge sampling
The three steps in the 3D-2D alignment are as follows:
1. Extract possible observable reflectance edges on 3D
reflectance images and edges along the occluding
boundaries based on the current viewing direction.
2. Establish correspondences between 3D edges and 2D
intensity edges.
3. Determine the relative relationship iteratively basedon the correspondences.

3.1

Visibility calculation and correspondences

Applying Canny operators to the reflectance images enables us to obtain reflectance edges. These are then sampled to edge sampling points, and stored on the surface ofthe 3D geometric model. First, the system decides whichedge part is visible from the current viewing direction. Visibility of an edge point,  , can be examined as

 



             (1)

where  is the surface normal of the patch and  is thecurrent camera’s viewing direction with respect to the current 3D geometric model. The edges along the occluding
boundary will also be sampled into edge points along theoccluding boundary, which is detected as

 



               (2)where is a threshold. Pruning of small edges is necessary. Currently, by adjusting various parameters in edge
tracking, we remove small edges and extract only dominant edges.

Correspondences

Pose estimation by M-estimator

To determine the relative pose that coincides the position of
2D intensity edges and 3D edges, we use the M-estimator,
which is one of the robust estimators.
First, the distance between corresponding 2D intensity
edge points and 3D edge points is evaluated as shown in
Fig.2: where  is a 3D error vector which is on a perpendicular line from a 3D edge point to the stretched line between the optical center and a 2D edge point on the image
plane.
    (3)

where  is the distance between the optical center and 3Dedge point, and  is the angle between 2D intensity edge
point and 3D edge point.
The system finds the configuration,  , which minimizes
the total error,  ,    

(4)

where  is an error function. The minimum of    canbe obtained by:     

We can consider
terms.

(5)  as a weight function to evaluate error    
(6)
 

Thus, the minimization can be derived as the following
least squared equation:

 


  




(7)

We choose the Lorentzian function for this function.

 ¾  ½
     (8)

By solving this equation using the conjugate gradientmethod, we can obtain the configuration  that minimizes
the error term and gives the relative relationship between
the camera and range sensor.

3D edge
2D edgeZ(x,y,z)2D Color Edge
θ(1) Initial position(2) After 1 iteration(3) After 2 iterations(4) After 5 iterationsdy

3D Edgeu

Projection of 3D Edge

Figure 2: 2D distance and 3D distance
Figure 4: Simulation results

4

Experiment

4.1

Computer simulation

In order to verify the performance of the proposed 2D-3Dalignment method, the computer simulation was carriedout. In this simulation, we assumed that a cylinder with
100 mm diameter and 200 mm length was placed 1000 mm
ahead of the color and range sensors. We did not assume
the reflectance values obtained from the range sensor, andused 3D edge points along the occluding boundaries only.
The resolution of the color sensor was 300 dpi and focus
length was 70 mm.
Fig.3 shows the simulation model, and Fig.4 shows the process of the alignment.

yxzFigure 3: Simulation model
From several initial poses with maximum 50 mm and 20
degrees difference from the actual pose, the optimum pose
was calculated. Table 1 shows the obtained average and thestandard deviation of the estimated position and the inclination of the y axis after 10 simulations.

In Table 1, the depth of the z axis is calculated closer than
the actual position. This is because 3D edge points along
the occluding boundaries are obtained as the center of triangle patches which satisfy Eq.2, and these center points
are calculated inside the actual boundaries. Thus, if the triangle patches were to be set more densely, this would not
be a problem.
As a result of this simulation, it wass verified that the accurate relative pose between the camera and the range sensor can be estimated by the proposed 3D-2D alignment
method.

4.2

Texture mapping of a dish with paintings

Next, using the Cyrax laser scanner , we attempted the texture mapping of a dish with paintings.
Fig.5 shows the reflectance image and extracted reflectanceedges, and Fig.6 shows the color texture image taken by
the digital camera (Nikon, D1) along with the extracted
intensity edges. Here, as shown in Fig.5, the brighter the
pixel color is, the larger its reflectance.
Fig.7 shows the procedure of the alignment of the reflectance and intensity images through the use of the Mestimation method. For these symmetrical objects, accurate alignment which coincides with the position of
the painting is impossible for the conventional methods
which use the geometric edges extracted from the geometric model or the CAD model. On the other hand, our proposed method can perform accurate alignment due to theuse of reflectance information obtained from the painting
as shown in Fig.8

Table 1: Position errors [mm (pixel)]

Average
STD.

x
0.14 (0.12)
0.13 (0.11)

y-0.20 (0.16)
1.89 (1.56)

z-967.81
5.94

[deg.]
4.0
4.1

4.3

Texture mapping of Great Buddha of
Kamakura

We have recently developed a method for preserving
Japanese cultural heritage objects. Using accurate laser

Figure 5: Reflectance image of the dishFigure 8: Aligned color texture on the dishFigure 6: Texture image of the dishscanners [9], 3D digital models of these objects are createdthrough observation. For our first preservation project, wehave created a 3D geometric model of the Great Buddha of
Kamakura as shown in Fig.9.
Using our proposed alignment method, we tried to map
the color texture image taken from the digital camera onto
this model. Fig.10 shows the reflectance image and extracted reflectance edges, Fig.11 shows the color texture
image taken by the digital camera. Fig.12 shows the procedure for the alignment of the reflectance and intensity
images through the use of the M-estimation method.
By comparing Fig.10 and Fig.11, the reflectance and thecolor image have robust similarities, for example, theshapes of rust on the forehead or the junctures of bronze.

Figure 9: Geometric model of the Great Buddha of Kamakura
Fig.13 shows the result of the mapping of the current color
texture onto the 3D geometric model. Fig.14 shows otherresults of applying the proposed alignment method.

Figure 10: Reflectance image of the Great Buddha of Kamakura

5 Conclusions

Figure 7: Intensity edges aligned with reflectance edges

In this paper, we proposed a new calibration method for
texture mapping through the use of reflectance images,
which are given as side products of range images for mostrange sensors. In this method, 3D reflectance edge points
and 2D color edge points are first extracted from the reflectance image and the color image, respectively. Then,the relative pose of each sensor is determined so that the 3D

Figure 11: Texture image of the Great Buddha of Kamakura

Figure 13: Aligned color texture on the 3D geometricmodel[2] Y. Sato, M. D. Wheeler, and K. Ikeuchi, ”Object
shape and reflectance modeling from observation”,
Proceedings of ACM SIGGRAPH 97, In Computer
Graphics Proceedings, Annual Conference Series
1997, ACM SIGGRAPH, pp.379-387, August 1997.
[3] K.Nishino, Y.Sato and K.Ikeuchi, ”Eigen-Texture
Method: Appearance Compression based on 3D
Model”, in Proc. of Computer Vision and Pattern
Recognition ’99, vol.1, pp.618-624, Jun., 1999.
[4] I. Sato, Y. Sato, and K. Ikeuchi, ”Acquiring a radiance distribution to superimpose virtual objects
onto a real scene,” IEEE Trans Visualization andComputer Graphics, Vol. 5, No. 1, pp.1-12, January
1999.
Figure 12: Aligned intensity edges with reflectance edges
reflectance edge points coincide with the 3D points calculated from the 2D color edge points be the use of the robust
M-estimator.
Using the proposed method, experiments of the texture
mapping for the 3D geometric model of Great Buddha of
Kamakura were carried out, and the usefulness of the proposed method was verified.
This project is funded by Core Research for Evolutional
Science and Technology (CREST) of Japan Science andTechnology Corporation (JST).

References[1] M. D. Wheeler, ”Automatic Modeling and Localization for Object Recognition”, Technical Report(Ph.D. Thesis), CMU-CS-96-188, School of Computer Science, Carnegie Mellon University, October,
1996.

[5] Mark D. Elstrom and Philip W. Smith,Stereo-Based
Registration of Multi-Sensor Imagery for Enhanced
Visualization of Remote Environments,Proceedingsof the 1999 IEEE International Conference on
Robotics & Automation, pp.1948-1953, 1999.
[6] Ioannis Stamos and Peter K. Allen, Integration ofRange and Image Sensing for Photorealistic 3D
Modeling, Proceedings of the 2000 IEEE International Conference on Robotics and Automation,
pp.1435-1440, 2000.
[7] P. Viola and W.M. Wells III, Alignment by maximization of mutual information, International Journal of Computer Vision, Vol.24, No.2, pp.137-154,
1997.
[8] M. D. Wheeler and Katsushi Ikeuchi, ”Sensor Modeling, Probabilistic Hypothesis Generation, and Robust Localization for Object Recognition”, IEEE
Transactions on Pattern Analysis and Machine Intelligence ,Vol. 17, No. 3, March 1995.

[9] Daisuke Miyazaki, Takeshi Ooishi, Taku
Nishikawa, Ryusuke Sagawa, Ko Nishino, Takashi
Tomomatsu, Yutaka Takase, Katsushi Ikeuchi,The
Great Buddha Project: Modelling Cultural Heritage
through Observation,VSMM2000 (6th international
conference on virtual systems and multimedia),pp.138-145, 2000.

Figure 14: The Great Buddha of Kamakura with the color texture image