ACCV2002: The 5th Asian Conference on Computer Vision, 23–25 January 2002, Melbourne, Australia.

1

Dense Estimation of Surface Reflectance Properties
for Merging Virtualized Objects into Real Images
Takashi Machida, Haruo Takemura and Naokazu YokoyaGraduate School of Information Science, Nara Institute of Science and Technology
E-mail: taka-ma, yokoya@is.aist-nara.ac.jp
Cybermedia Center, Osaka University
E-mail: takemura@cmc.osaka-u.ac.jp

Abstract
To appropriately reproduce a real object in a mixed environment, it is necessary to estimate reflectance propertiesof object surfaces. This paper describes a new method ofdensely estimating non-uniform surface reflectance properties of an object with convex and concave surfaces using
registered range and surface color texture images obtainedby a laser rangefinder. The proposed method determines
positions of light source to take color images for discriminating diffuse and specular reflection components of surface reflection. The Torrance-Sparrow model is employed
to estimate reflectance parameters using color images under multiple illumination conditions. Experiments show theusefulness of the method.

1

Introduction

In mixed reality (MR), it is needed that a virtualized object is merged in real world without a feeling of wrongness[1]. To represent a virtualized object photo-realistically,
there are two approaches. One is called an image based
rendering (IBR) that has often been used to reproduce real
objects in computer graphics (CG). Generally, IBR methods
require a large number of real images to represent the virtualized object under arbitrary illumination conditions andarbitrary viewing directions. Mukaigawa et al. [10] haveproposed a photometric IBR in which a virtualized object
is represented with a few real images. However, in the caseof using a limited number of images, a problem occurs; i.e.,
the appearance of object is not faithfully reproduced sincesome part of the object is interpolated linearly. The othermethod is called a polygon based rendering (PBR). Thismethod reproduces the object shape and surface reflectanceproperties. If the object surface reflectance properties are
estimated at once, the virtualized object can be renderedappropriately under virtualized illumination conditions es-

timated from real environments [11, 3]. A number of methods of estimating reflectance properties of an object surface
have been developed [2, 5, 6, 7, 8, 9, 12]. In these methods, surface reflectance models with several parameters are
employed and shape and color information of the object isused to estimate the reflectance parameters.
In some works [2, 5, 9], it is assumed that an object has auniform reflectance property over the entire surface. Reflectance parameters are estimated by using the standard
least-squares method to fit a reflectance model to a given
color image. Due to the assumption, such methods cannot be applied to objects which consist of several different
materials and have non-uniform reflectance properties. To
treat non-uniform surface objects, Kay and Caelli [6] haveproposed a method which uses multiple images of an object
under different lighting conditions and estimates reflectanceparameters by solving simultaneous equations. However,
the method still has a problem that results are not stable especially when the specular reflection component, which isone of the reflection components, is very small.
Recently, Sato et al. [12] have developed a methodology
to estimate non-uniform reflectance properties. They set up
an object on a robot arm and measure the object with a CCD
camera and a rangefinder from a large number of viewpoints
by rotating the robot arm. In the method, reflectance parameters are stably acquired by decomposing the surface reflection into two components based on the singular value decomposition (SVD) technique. Although the method can beapplied to objects with non-uniform reflectance properties,
the shape of object should be limited. This is because it isdifficult to observe the specular reflection component overthe entire surface, since the lighting condition for a poseof the object against the camera cannot be changed in themethod.
We propose a new method for estimating non-uniform
reflectance properties by observing the specular reflection component densely in the image obtained by a laser

Laser rangefinder

A. Measurement
Range data
B. Calculation of surface normal
Possible light position 1
Ratio of observing eachreflection component. . . .

ObjectPossible light position M
Ratio of observing eachreflection component

C. Selection of light source position(a) Appearance of
3D-Digitizer

D. Measurement
Selected light positoin 1
Surface texture image. . . .

Selected light position m
Surface texture image

E. Estimation of reflectance property

Center of rotation

Figure 1. Flow diagram of estimating surface
reflectance properties.

Figure 2. 3D-Digitizer.
an angle between a viewing vector Î and a surface normal
vector Æ, and  is an angle between a viewing vector Îand a reflection vector Ä¼ , where the reflection vector is thevector which Ä is mirrored against Æ. All vectors are normalized as unit vectors.

Estimation of reflectance parameters fromrange and color images

2.2

In this paper, the Torrance-Sparrow model [13], which
physically represents object reflectance properties, is employed to estimate reflectance parameters. The TorranceSparrow model is given as:










      Measurement of an object

We use a laser rangefinder (Cyberware 3030RGB) withknown positions of point light sources and a camera for acquiring surface color images, as illustrated in Figure 2(a).
Figure 2(b) shows its motion during measurement. Thissystem can obtain registered range and surface color texture images at the same time by rotating the rangefinder andthe camera around an object, so that there is no registration
error, even when an object is measured many times. Figure 2(c) shows the illustration viewed from the top of thedevice. A camera is located atand a texture image isacquired through mirrors which are located at  and.
We assume that the camera is located at
virtually andthe camera looks toward the center of rotation.

Torrance-Sparrow model

Virtual camera
X4(c) Direction of camera

In the present work, the Torrance-Sparrow model is employed as a surface reflectance model to estimate object surface reflectance properties from range and color images.
Figure 1 shows a flow diagram of estimating surface reflectance properties. Our process consists of four parts,
which are a measurement of an object (A, D), a preprocessing (B), a selection of light source (C), and an estimation ofreflectance parameters (E).

2.1

Laser rangefinder
X1
X2
CameraX3

rangefinder which accurately takes registered range andcolor images of an object. In this paper, an algorithm is proposed to determine light positions with which the specularreflection component is strongly observed over the surface
of a complex object with convex and concave surfaces.

2(b) Motion during
measurement

2.3

(1)

Preprocessing

Generally, the noise and quantization error are includedin the range image acquired from the laser rangefinder.
There is also a problem that the surface normal is not calculated accurately around the discontinuity in the range image.
Therefore, we employ an adaptive local quadratic surface
fitting.
Firstly, the 55 median filter is applied to the range image to remove the noise. Secondly, the quadratic surface iswhere  represents an observed intensity,  is an attenuation coefficient concerning the distance between a point
light source and an object surface point,  represents thestrength of a light souce.   is the diffuse reflectance parameter,  is the specular reflectance parameter, is thesurface roughness parameter which is the standard deviation of a Gaussian distribution,   is an angle between alight source vector Ä and a surface normal vector Æ,  is2

Laser rangefinder

locally fitted with the range image. The range image fromthe Cyberware rangefinder is expressed by the cylindricalcoordinates. Each point of the range image is representedas:Ü Ý Þ               (2)Possible light position

where , ,  are the distance from the center of rotation,
the angle of rotation, and the height along the rotation axis,
respectively.
The unit normal vector is given as follows:Æ 

Figure 3. Multiple possible light source positions.

              




 
    

is calibrated based on the distance from the center of rotation in world coordinates. When optimum light positions
are selected, a single light is attached at selected possible
positions in turn. Therefore, the calibration of brightness
among multiple lights is not needed.
Let  be a color image which is to be obtained with apossible light position        and consists of pixels      , where  means a color intensity,
 be the number of pixels which include only the diffuse
reflection component in   , and  be the number of pixels
which include the specular reflection component strongly in .
First, the following conditions are examined for eachpixel in the object surface texture under each light position.

   (3)where  ,  are gradient components of the range image
 . This gradient is computed using the following local
surface fit:¼

               (4)

Coefficients    are determined by minimizing thefollowing equation using the range data   and Equation
(4).




            

 Measurability of light reflection
 Measurability of only diffuse reflection
 Measurability of strong specular reflection      (5)where ,  are local coordinates in a    window. In our¼

Second, the light positions  and  which satisfy          and          are selected. In the next light source position selection, the position which satisfies the same condition is selected among
the rest except for light source positions decided so far.
Then,  light positions are selected to densely estimate
reflectance parameters. The selections of positions are repeated until almost all pixels are observed once for only the
diffuse reflection component and twice for the strong specular reflection component.
We introduce a threshold , i.e., the ratio of the measurability of both reflection components to stop the processof selecting light positions. With respect to determining thethreshold , we judge the ratio of the measurability of the
specular reflection with all possible positions. This ratio
is a limit of measuring the specular reflection component.
Using the threshold, we can terminate the light selection
process in the case that the ratio of measurability of specular reflection component cannot increase more, even if thenumber of positions of light source is increased. In sucha way, reflectance parameters can be efficiently estimatedalmost the whole object surface using a limited number of
texture images.

approach, selected local quadratic surface fit is achieved byusing the Yokoya-Levine operator [14]. In [14], the best
window is selected among 25 possible windows which include the point   to estimate the coefficients    at
 . The best window provides the minimum fitting error
in Equation (5).

2.4

Selection of light source

To densely estimate non-uniform reflectance parameters
independently, it is required to observe each pixel under
three different lighting conditions: One for observing only
the diffuse reflection component to determine one unknown
parameter  and others for observing both the diffuse andspecular reflection components to acquire two unknown parameters  and . Therefore, in the present experimental
setup, multiple positions of a light are determined among
60 possible positions prepared around the laser rangefinder
and these are arranged with 5 vertically and with 12 horizontally at the interval of 5 cm as shown in Figure 3. The
position of a camera is calibrated in advance by measuring
a box whose size is known. The position of a light souce3

L

2.4.3 Measurability of strong specular reflection

N

L’

When -th pixel includes the specular reflection strongly,
the reflection vector Ä¼ satisfies the following equation.

V
θr

θ th1

Range of observing
both reflection
components

Object surface

Range of observing
only the diffuse
reflection component



V

L
N
L’

θth2



Î  Ä    ¼(8)

where   is a threshold angle between Î and Ä¼ . Equation (8) means that both the diffuse and specular reflection
components are observed if  is smaller than   as illustrated in Figure 5. The above condition is based on the factthat the specular reflection is observed strongly in a limited range of a viewing angle. When this condition stands
and the pixel is not in a shadow, the pixel is judged to havestrong specular reflection and is counted in   .

Figure 4. Observation of only the diffuse reflection component.

Range of observing
the specular reflection
component strongly

θr

 

Object surface

2.5
Figure 5. Observation of the specular reflection component strongly.

Let  
be the set of pixels which are judged to consist of only the diffuse reflection component with the light
position  and consist of ! pixels           ,
where  means a color intensity. Let   be
the set of pixels which are judged to include the specular reflection component strongly and consist of " pixels
         , where   means a color intensity. Note that there are some pixels which consist ofneither the diffuse reflection component only, nor the strong
specular reflection component in  and   . With
respect to such a pixel, reflectance parameters cannot be estimated.
The diffuse reflectance parameter  at the -th pixel
is estimated from the value of   in the image   ,Æ and Ä . In order to get the most reliable estimation, thepixel whose angle  is the smallest but greater than   isselected.
The specular reflectance parameter  and the surface
roughness parameterat the -th pixel are estimated bysolving a simultaneous equation of Equation (1) with thevalue of the specular reflection component extracted fromthe -th pixels    and   in the images  and   , Æ , Ä , Î , and  estimated above. In
order to get the most reliable estimation, the pixels whoseangle  is the smallest or the second smallest and is smaller
than  , are selected.

2.4.1 Measurability of light reflection
In order to measure the light reflection at a specific point of
the object surface, the point on the surface must be observable from the camera position. Additionally positional relationship among the camera, the point and the light sourcemust satisfy the following conditions.

Î  Æ   ¼ Ä  Æ   ¼(6)

where Î , Ä , and Æ are the viewing direction, the lightsource direction, and the surface normal at the -th pixel,
respectively. Note that the viewing direction Î  and thesurface normal Æ are independent of the light source position . Even when the above equations are both satisfied,there is a possibility that a shadow is casted on the pixel.
In other words, a point on the object may be covered by ashadow casted by light source . This can be judged by anexisting technique[4].
2.4.2 Measurability of diffuse reflection only
When the -th pixel consists of only the diffuse reflection,
the reflection vector Ä¼ satisfies the following equation.


 



Î  Ä    ¼

Estimation of reflectance parameters

3(7)

where   is a threshold angle between Î and Ä¼ .
Equation (7) implies that only the diffuse reflection component is observed if   is greater than   as illustrated in
Figure 4. When this condition stands and the pixel is not ina shadow, the pixel is judged to have diffuse reflection only
and is counted in   .

Experiments

In our experiments, a measured object is a plastic dolland exibits non-uniform surface reflectance as shown in
Figure 6(a) and is assumed not to have interreflections. We
fix some parameters as    ,    ,   .
The proposed method of selecting positions of the lightsource determined 12 light source positions for the object.
4

(b) Diffuse reflectance parameter  (d) Surface roughness parameter  (c) Specular reflectance parameter   (e) Area of observing specular reflection(a) Object













%

%

Ratio of measurabilityof specular reflectance andsurface roughness
parameters

Ratio of measurability ofdiffuse reflectance parameter

Figure 6. A measured object and estimated reflectance parameters.















        



















        

Number of selected light source positions

Number of selected light source positions(a) Possibility of estimating the diffuse reflectanceparameter













(b) Possibility of estimating the specular reflectance and thesurface roughness parameter

Figure 7. The possibility of estimating each reflectance parameter with respect to the number of light
positions.
Figure 7 illustrates the measurability of both reflection
components with respect to the number of light sources.
Note that the horizontal axis is represented until 20, sincethese graphs are stable even if all possible positions areused. In Figure 7(a), with respect to the diffuse reflectanceparameter, when the number of selected light source positions is 5, the ratio of the measurability of the diffuse reflection component is . In Figure 7(b), as for the specular reflectance and the surface roughness parameters, evenwhen the object is measured with all possible positions, theratio of the measurability of the specular reflection component is  # . In our method, the ratio of measurability ofthe specular reflection component is  #  with automatically selected 12 light source positions. Since this figure iscomparable to that with all possible light source positions,the experiment shows that the proposed method is efficient.

Figure 6(b) illustrates the diffuse reflectance parameter
estimated all over the object surface. Figure 6(c) is the specular reflectance parameter. This image clearly shows thatthe specular reflectance parameter of the doll’s beak andleg are different from the rest. Actually, the beak and legs
are highly reflective as in Figure 6(a). Figure 6(d) shows
the surface roughness parameter with gray-scale where thelargest value is coded as white. This image means thatthe smaller the value is, the smoother the object surface
is. Figure 6(e) shows the ratio of pixels where specular reflectance and surface roughness parameters are computed.
The black part means that both parameters are not directly
estimated. Non-uniform specular reflectance and surface
roughness parameters were estimated without interpolation
for 83.46% of the surface. Note that there is a part wherereflectance parameters cannot be estimated. These parts arelinearly interpolated.

Figure 8(b) and (c) show results of merging the virtual5

(a) geodesic dome(c) result 2(b) result 1
Figure 8. Merging a virtualized object into a real image.

ized object into a real image. The real illumination condition is modeled by light sources on a geodesic dome surrounding the object as shown in Figure 8(a). Virtual lightsources are uniformly spaced on the geodesic dome andthese brightness are the same. The number of virtual lights
is 42. Because surface reflectance parameters are estimateddensely, the virtualized object is represented without a feeling of wrongness. Frame rate is 3.2 fps on a Pentium III
1GHz PC. Therefore, if an efficient rendering method isachieved, the virtualized object can be merged into realworld in real time.

4[2] R. Baribeau, M. Rioux, and G. Godin. Color reflectancemodeling using a polychromatic laser sensor. IEEE Trans.
on Pattern Anal. Mach. Intell., 14(2):263–269, 1992.
[3] P. Debevec. Rendering synthetic objects into real scenes:
Bridging traditional and image-based graphics with global
illumination and high dynamic range photography. Proc.
SIGGRAPH ’98, pages 189–198, 1998.
[4] J. Foley, A. van Dam, S. Feiner, and J. Hughes. Computer
Graphics Principles and Practice. Addison-Wesley Publishing Company, 2nd edition, 1993.
[5] K. Ikeuchi and K. Sato. Determining reflectance propertiesof an object using range and brightness images. IEEE Trans.
on Pattern Anal. Mach. Intell., 13(11):1139–1153, 1991.
[6] G. Kay and T. Caelli. Inverting an illumination model fromrange and intensity maps. CVGIP: Image Understanding,
59:183–201, 1994.
[7] S. Lin and S. W. Lee. Estimation of diffuse and specularappearance. Proc. Int. Conf. on Computer Vision, 2:855–
860, 1999.
[8] S. Lin and S. W. Lee. A representation of specular appearance. Proc. Int. Conf. on Computer Vision, 2:849–854, 1999.
[9] J. Lu and J. Little. Reflectance function estimation and shaperecovery from image sequence of a rotating object. Proc. Int.
Conf. on Computer Vision, pages 80–86, June 1995.
[10] Y. Mukaigawa, H. Miyaki, S. Mihashi, and T. Shakunaga.
Photometric image-based rendering for image generation inarbitrary illumination. Proc. Int. Conf. on Computer Vision,
pages 643–649, July 2001.
[11] I. Sato, Y. Sato, and K. Ikeuchi. Acquiring a radiance distribution to superimpose virtual objects onto a real scene. IEEE
Trans. on Visualization and Computer Graphics, 5(1):1–12,
January 1999.
[12] Y. Sato, M. D. Wheeler, and K. Ikeuchi. Object shape andreflectance modeling from observation. Proc. SIGGRAPH’97, pages 379–387, 1997.
[13] K. E. Torrance and E. M. Sparrow. Theory for off-specularreflection from roughened surfaces. Journal of Optical Society of America, pages 1105–1114, 1967.
[14] N. Yokoya and M. D. Levine. Range image segmentation based on differential geometry: A hybrid approach.
IEEE Trans. Pattern Anal. Mach. Intell., 11(6):643–649,
June 1989.

Conclusions

In this paper, we have proposed a new method of densely
estimating non-uniform reflectance properties for almost
the whole object surface by using the laser rangefinder for
virtualizing real objects. In our approach, multiple lightsource positions around the laser rangefinder are automatically selected, so that diffuse and specular reflection components are observed densely. The experiments have shownthat the proposed method is useful for estimating the reflectance parameters of objects which exhibit non-uniform
surface reflectance. In the future, we will investigate amethod to consider interreflections all over the object surface. Moreover, we have to automatically estimate real illumination conditions for merging virtualized objects into
real images for mixed reality applications.

Acknowledgments: This work was supported in partby the Grant-in-Aid for Scientific Research under the contract no.13480098 from the ministry of Education, Science,Sports and Culture.

References[1] R. T. Azuma. A survey of augmented reality. Presence,
6(4):355–385, 1997.

6