VIRTUALIZING REAL OBJECTS WITH SURFACE INTERREFLECTIONSTakashi Machida
Cybermedia Center, Osaka University
1-30 Machikaneyama, Toyonaka, Osaka
E-mail: machida@ime.cmc.osaka-u.ac.jp

ABSTRACTIn mixed reality, it is important to estimate object surface
reflectance properties to render real objects under arbitrary
illumination conditions. The authors developed a method toestimate reflectance properties of object surfaces densely.
However, it was difficult to estimate surface reflectanceproperties faithfully for objects with interreflections. Thispaper describes a new method of densely estimating nonuniform surface reflectance properties of real objects constructed of convex and concave surfaces with interreflections. The proposed method first determines positions oflight to take color images for discriminating diffuse andspecular reflection components of surface reflection. Then,surface reflectance parameters are estimated based on radiosity. Experiments show the usefulness of the proposedmethod.
1. INTRODUCTION
In constructing a mixed reality environment, it is required
to obtain surface reflectance properties of virtualized realobjects. This could make it possible to merge virtualized
objects seamlessly into a virtual world under arbitrary illumination conditions. For this purpose, we have conducteda research to estimate surface reflectance properties denselyfrom range and surface texture images [1].
Object surface reflectance properties consist of diffuse
and specular components. The diffuse reflection component
is easily observed due to its nature of reflection in omnidirection. On the other hand, it is difficult to observe thespecular reflection component due to its nature of reflection
within a fixed angle with respect to a viewing position, lightsource and object surface normal vector. Moreover, even if
the specular reflection component is observed, this component cannot be estimated stably due to small values. Our resent research has solved this problem by selecting optimum
light positions to observe both components [1]. However,there still exists a problem such that interreflections are notconsidered.

Naokazu Yokoya
Graduate School of Information Science,
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara
E-mail: yokoya@is.aist-nara.ac.jp
In this paper, we take notice of the radiosity renderingmethod which can represent interreflections based on thetransmission of radiosity (heat energy of light). Fournier etal. [2] has originally developed a method to estimate surface reflectance properties using the radiosity method in anindoor scene. This method assumes uniform reflectance on
surface and determines only the diffuse reflectance parameter. There are some attempts to estimate both surface reflectance properties of a room [3, 4]. They, however, also
assume that almost all objects have the same reflectanceproperties, therefore, their algorithm cannot be applied toan object which has non-uniform surface reflectance properties.
This paper proposes a new method for estimating nonuniform reflectance properties of objects with interreflections to represent virtualized objects photo-realistically. The
proposed method can observe the specular reflection component densely considering interreflections based on radiosity.
2. ESTIMATION OF REFLECTANCEPARAMETERS FROM RANGE AND COLOR
IMAGES
Figure 1 shows a flow diagram of densely estimating surface
reflectance properties of real 3D objects. Our process canbe divided into three parts: Measurement of object (A, C),selection of light source (B), and estimation of reflectanceparameters (D).
2.1. Measurement of object
We use a laser rangefinder (Cyberware 3030RGB) withknown positions of point light sources and a camera for
acquiring surface color images, which is illustrated in Figure 2 (a). This system can obtain registered range and surface color texture images at the same time by rotating therangefinder and the camera around an object, so that there
is no registration error, even when the object is measured
many times.

tor. Note that the reflection vector is a vector to which alight vector is mirrored against a normal vector.
It is required to observe each surface point under threedifferent lighting conditions: One for determining   andothers for acquiring   and . Here, if the value of  issmall, the specular reflection component is judged to bereduced seriously. Therefore, we can use the value of to judge whether the specular reflection component is observed or not. The selection of position of a light source isrepeated until almost all surface points satisfies three different lighting conditions (See [1] for detail). Then, a certain
number of light positions, say , are selected to densely
observe both reflection components.

A. Measurement
Range data
Possible light position 1
Ratio of observing eachreflection component. . . .

Possible light position M
Ratio of observing eachreflection component

B. Selection of light source position
C. Measurement
Selected light positoin 1
Surface texture image. . . .

Selected light position m
Surface texture image

D. Estimation of diffuse reflectance parameter based on radiosity
Estimation of specular and surface roughness parameters

Figure 1: Flow diagram of estimating surface reflectanceproperties.
Laser rangefinder

2.3. Estimation of reflectance parameters
A texture image is obtained with a selected light position
pixels  ½    ,where  means a color intensity. Each pixel is classified
into three types  ,  and   .  means a pixel
containing only the diffuse reflection component and means a pixel containing strong specular reflection component.   means a pixel classified into neither  nor
 and such a pixel is not used to estimate reflectanceparameters.
        and consists of: Possible light position(a) Appearance(b) 60 possible light positionsof 3D-Digitizer
Figure 2: 3D-Digitizer and experimental setup.

2.3.1. Estimation of diffuse reflectance parameter basedon radiosity

2.2. Selection of positions of light source
In the present experimental setup, multiple positions ofa light are determined among 60 possible positions prepared around the laser rangefinder and these are twodimensionally arranged at the interval of 5 cm as shown in
Figure 2 (b). The positions of a camera and a light source
are calibrated in advance. When optimum light positions
are selected, a single light is attached at the selected positions in turn. Therefore, the calibration of brightness among
multiple lights is not needed.
Here, we employ the Torrance-Sparrow model [5] torepresent object reflectance properties physically. The
Torrance-Sparrow model is given as:

      (1)

where  represents an observed intensity,  is an attenuation coefficient concerning the distance between a point
light source and an object surface point,  represents thestrength of a light source.   ,  and  are the diffuse
reflectance, the specular reflectance and the surface roughness parameters, respectively.  is an angle between a lightsource vector and a surface normal vector,  is an angle between a viewing vector and a surface normal vector, and is an angle between a viewing vector and a reflection vec-We assume in this paper that interreflections do not havean influence from both reflection components but only fromthe diffuse reflection component. Therefore, a pixel whichis categorized into  is used in the following method.
In this paper, to consider interreflections, we employ theradiosity equation given as:   (2) whereand      represent the radiosity of -th and -th patch, respectively. Note that  is thenumber of object patches.   is a form factor between thepatch and the patch .  is an environment term which
represents an influence on the patch . In this paper, we assume that   , because the object is measured in a darkroom.  represents the diffuse reflectance parameter on
the patch .
In the proposed method, the form factor   is known because the object shape is measured by the laser rangefinder.
Since the range and texture images are registered at eachsurface point, the radiosityof the patch is calculated
based on the value of the surface point which corresponds tothe patch . Then, the diffuse reflectance parameter  isestimated by solving Eq. (). Finally, the diffuse reflectanceLaser rangefinderp(t)ObjectRegion A

s u t
p(s)

Region B(b) Object B(a) Object A(c) Object C
Figure 4: Three objects used in experiment.

Figure 3: Calculation of radiosity.
parameter at a each surface point is estimated by calculating the average among neighbouring patches which sharethe point.
is described. LetHere, the calculation of radiosity be the position of a light source when a surface point is measured as shown in Figure 3, and  be the value of theof the patch which insurface point . The radiosity
cludes surface points  and  is calculated as follows. Each
patch consists of four points andis represented as thesum of the values of the corresponding pixels. Because alight source attached with the laser rangefinder moves during measurement, it is necessary to calculate  with thelight position .   is interpolated linearly by the following equation since   and  are categorized in  .
        


  



  

(3)

where  and  are normal vectors at surface points  and.
´µ and ´µ are light vectors with the light position at surface points  and . Note that in the case of ,  should be 0 to consider the measurability of lightreflection.



2.3.2. Estimation of specular reflectance and surface
roughness parameters based on Torrance-Sparrow model
The specular reflectance parameter  and the surface
roughness parameter  at the -th pixel which is categorized in  are estimated by solving a simultaneous
equation of Eq. (1) with the values of the specular reflection component extracted from two images taken under two
different illumination conditions and   estimated above.
See [1] for detail. Note that   estimated above should bescaled to Eq. (1) before computing the specular and surface
roughness parameters.
3. EXPERIMENTS
We show that the proposed method can remove an influenceof interreflections in a preliminary experiment. We separate the object A shown in Figure 4 (a) into two regions
A and B. We have conducted two preliminary experiments.
One is that the same white paper with a uniform diffuse reflectance is pasted up on both regions. The other is that theTable1: Measurability of two reflection components.
Measurment object
The number of selected light positions
Diffuse reflection component
Specular reflection componentB
9(3)100.0%
94.7%

C13(5)
100.0%
85.9%

same glossy paper with a uniform reflectance is pasted up
on both regions. The proposed method is compared withthe conventional method [1]. Results are shown in Figures
5 and 6. Each graph represents RGB channels of the diffuse
reflectance parameter estimated by both methods. The horizontal axis means the position of the surface point in thevertical direction of the object and the vertical axis means
the average of diffuse reflectance parameters in the horizontal direction of the object. In Figure 5, in the conventional method, the value of the diffuse reflectance parameter is large around the boundary between the regions A andB because the conventional method does not consider interreflections. On the other hand, such an effect is eliminatedin the proposed method. This clearly shows the usefulnessof the proposed method. However, the influence of interreflections still remains in the glossy surface as shown in
Figure 6, because interreflections due to the specular reflection also ocurrs at the boundary between the regions A andB in the second preliminary experiment.
In the next experiment, We show that the proposed
method is useful for real objects with complex shapes: Objects B and C shown in Figures 4 (b) and (c). The measurability of both reflection components and the number of
selected light sources are shown in Table 1. The numberin the bracket means the number of selected positions of alight source required to estimate the diffuse reflectance parameter. From these results, it is observed that non-uniform
surface reflectance properties are estimated over almost thewhole surface considering interreflections with a limited
number of light positions.
Figure 7 shows rendering results of virtualized objects Band C with multiple viewing angles, respectively. A virtual
light source is located at a fixed position. In rendering, direct reflection from the virtual light source is represented byusing the Torrance-Sparrow model. Each object is rendered
suitably even at the part which contains interreflections dueto the diffuse reflection component. However, in the case130120110

Proposed method

Region A

Region B

100
1214161

81 101 121 141 161 181 201 221

150

Conventional method

140130120110

Proposed method

Region A

Region B100
1214161

81 101 121 141 161 181 201 221

The average of the diffuse
reflectance parameter

Conventional method

140

The average of the diffuse
reflectance parameter

The average of the diffuse
reflectance parameter

150150Conventional method

140130120110

Proposed method
121T he posit ion of pixel in vertical direct ion

T he posit ion of pixel in vert ical direction

4161

Region B

81 101 121 141 161 181 201 221

T he position of pixel in vert ical direction(b) G channel(a) R channel

Region A100(c) B channel

Conventional method

140130120
110

Proposed method

Region ARegion B100
1214161

81 101 121 141 161 181 201 221150

Conventional method

140130120110

Proposed method

Region ARegion B100
1214161

81 101 121 141 161 181 201 221

The average of the diffuse
reflectance parameter

150

The average of the diffuse
reflectance parameter

The average of the diffuse
reflectance parameter

Figure 5: A comparison with previous work for a diffuse surface.
150

Conventional method

140
130
120
110

Proposed method
1

21

T he posit ion of pixel in vertical direct ion

T he posit ion of pixel in vert ical direction

Region B

4161

81 101 121 141 161 181 201 221

T he position of pixel in vert ical direction(b) G channel(a) R channel

Region A100(c) B channel

Figure 6: A comparison with previous work for a glossy surface.

LeftFront(a) Object B

RightFront(b) Object C
Figure 7: Rendering of virtualized objects with Torrance-Sparrow model.

of object C which has a complicated shape, interreflections
due to the specular reflection component cause some errors.
It is necessary to eliminate such an influence.

4. CONCLUSIONSIn this paper, we have proposed a new method of densely estimating non-uniform reflectance properties of real objects
with interreflections. In the proposed approach, we haveestimated surface reflectance properties based on radiosity.
The experiments have shown that the proposed method isuseful for estimating the reflectance parameters of objects
which exhibit non-uniform surface reflectance. Some extensions for considering specular interreflections should further
be investigated.

5. ACKNOWLEDGMENTS
This work was supported in part by Grant-in-Aid for Scientific Research under Grant No. 13480098 from the Ministryof Education, Culture, Sports, Science and Technology.

LeftRight

6. REFERENCES[1] T. Machida, H. Takemura, and N. Yokoya, “Dense estimation of surface reflectance properties for merging
virtualized objects into real images,” Proc. ACCV 2002,pp. 688–693, January 2002.
[2] A.Fournier, A.S. Gunawan, and C. Romanzin, “Interactive common illumination for computer augmentedreality,” Proc. Graphics Interface 1993, pp. 254–262,
May 1993.
[3] C. Loscos, G. Drettakis, and L. Robert, “Interactive virtual relighting of real scenes,” IEEE Trans. Visualization and Computer Graphics, vol. 6, no. 4, pp. 289–305,
December 2000.
[4] Y. Yu, P. E. Debevec, J. Malik, and T. Hawkins, “Inverse
global illumination: Recovering reflectance models ofreal scenes from photographs,” Proc. SIGGRAPH ’99,pp. 215–227, August 1999.
[5] K. E. Torrance and E. M. Sparrow, “Theory for offspecular reflection from roughened surfaces,” Journalof Optical Society of America, pp. 1105–1114, 1967.