Ph. D. Dissertation

Dense Estimation of Surface Reflectance Properties
Based on Inverse Rendering

TAKASHI MACHIDA

March 17, 2005

Department of Information Systems
Graduate School of Information Science
Nara Institute of Science and Technology

Ph. D. Dissertation
submitted to Graduate School of Information Science,
Nara Institute of Science and Technology
in partial fulfillment of the requirements for the degree of
DOCTOR of ENGINEERING

TAKASHI MACHIDA
Thesis committee: Naokazu Yokoya, Professor
Kunihiro Chihara, Professor
Yoshitsugu Manabe, Associate Professor

Dense Estimation of Surface Reflectance Properties
Based on Inverse Rendering 
TAKASHI MACHIDA

Abstract
In computer graphics (CG), making photorealistic images using a computer is now
commonplace. As a result, directors can create convincing, imaginary worlds, and designers can virtually prototype, visualize, and evaluate potential products and spaces.
In order to achieve these purposes, rendering methods such as ray tracing and radiosity
rendering methods have been developed. However, the rendering of more photorealistic images requires both accurate object surface reflectance parameters and object
surface geometries to be obtained. Therefore, in augmented virtuality, it is importantto estimate surface reflectometry and surface geometry from real objects or scenes. In
particular, object surface reflectometry estimation is of primary importance because,
unlike object surface geometry, which can be measured using a range finder, no device
has been developed to measure the variation of object surface reflectance properties.
The present study investigates the problem of object surface reflectance estimation, which is sometimes referred to as inverse reflectometry, for photorealistic rendering and effective multimedia applications. A number of methods have been developedfor estimating object surface reflectance properties in order to render real objects underarbitrary illumination conditions. However, it is difficult to densely estimate surface reflectance properties faithfully for complex objects with interreflections. This thesis describes three new methods for densely estimating the non-uniform surface reflectanceproperties of real objects constructed of convex and concave surfaces. Specifically, we
use registered range and surface color texture images obtained by a laser rangefinder.
The proposed methods determine the positions of light sources in order to capture color
 Ph. D. Dissertation, Department of Information Systems, Graduate School of Information Science,

Nara Institute of Science and Technology, March 17, 2005.

iimages to be used in discriminating diffuse and specular reflection components of surface reflection. The first method involves densely estimating local surface reflectanceproperties based on the calculation of an inverse local illumination rendering. Thismethod is useful only for objects without surface interreflections. The second andthird methods densely estimate the reflectance parameters of objects with diffuse andspecular interreflections based on an inverse global illumination rendering. Experiments are conducted in order to demonstrate the usefulness and the advantages anddisadvantages of the proposed methods through comparative study.

Keywords:
Inverse Rendering, Surface Reflectometry, Interreflections, Mixed Reality, Augmented
VirtualityiiAcknowledgements
This work was completed under the supervision of Professor Naokazu Yokoya of the
Graduate School of Information Science at the Nara Institute of Science and Technology and Professor Haruo Takemura of the Cybermedia Center at Osaka University.
I would especially like to thank my advisor, Professor Naokazu Yokoya, who has
been a great source of support and guidance throughout my Ph.D. study. Without
the continuous encouragement and advice of Professor Yokoya, I could never havecompleted this thesis. I would also like to thank Professor Haruo Takemura, who has
also been a wonderful source of support and guidance throughout the preparation ofthis thesis.
I am grateful to Professor Kunihiro Chihara and Associate Professor Yoshitsugu
Manabe, as members of the thesis committee, for their insightful comments on thisresearch and for accommodating a tight thesis defense schedule.
I have also benefited greatly through being able to meet and work with a number ofpeople during my doctoral work at the Nara Institute of Science and Technology. Associate Professor Kazumasa Yamazawa and Assistant Professors Hidehiko Iwasa (nowNet Systems Inc.), Masayuki Kanbara and Tomokazu Sato of the Graduate School ofInformation Science at the Nara Institute of Science and Technology provided helpfulcomments and invaluable discussions. I would also like to acknowledge the help andsupport that I received from the members of the Vision and Media Computing Laboratory of the Graduate School of Information Science at the Nara Institute of Scienceand Technology, especially my colleagues, Mr. Soichi Ebisuno, Mr. Jun Shimamura,
Mr. Hirofumi Fujii, and Mr. Koichiro Yamaguchi, for making life more pleasant andwork more fun during my Ph.D. study.
Finally, I would like to acknowledge the enormous support from my family and to
express my sincere thanks to my parents for their understanding and support.
iii

Contents
Abstracti

Acknowledgementsiii

List of Figuresix

List of Tables

xiii

List of Symbols

xv

1

Introduction1

2

Dense Observation of Reflection Components
2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2. 3D measurement of an object . . . . . . . . . . . . . . . . . . . . . .
2.3. Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4. Selection of light source positions based on the reflectance model . . .
2.4.1 Torrance-Sparrow model . . . . . . . . . . . . . . . . . . . .
2.4.2 Measurability of light reflection . . . . . . . . . . . . . . . .
2.4.3 Measurability of diffuse reflection only . . . . . . . . . . . .
2.4.4 Measurability of strong specular reflection . . . . . . . . . . .
2.5. Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11
11
13
15
17
18
22
24
24
26
32

3

Inverse Local Rendering for Dense Reflectometry Estimation
3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2. Estimation of surface reflectance parameters . . . . . . . . . . . . . .

33
33
34

v

3.2.1
3.2.2

Estimation of diffuse reflectance parameter . . . . . . . . . .
Estimation of specular reflectance and surface roughness parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3. Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1 Estimated reflectance parameters . . . . . . . . . . . . . . . .
3.3.2 Rendering results with the Torrance-Sparrow model . . . . .
3.4. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4

5Inverse Radiosity for Dense Reflectometry Estimation
4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2. Radiosity rendering method . . . . . . . . . . . . . . . . . . . . . . .
4.3. Estimation of surface reflectance parameters considering diffuse interreflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 Estimation of diffuse reflectance parameter based on radiosity
4.3.2 Estimation of specular reflectance and surface roughness parameters based on the Torrance-Sparrow model . . . . . . . .
4.4. Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.1 Estimated reflectance parameters . . . . . . . . . . . . . . . .
4.4.2 Rendering results using the radiosity method and the TorranceSparrow model . . . . . . . . . . . . . . . . . . . . . . . . .
4.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Inverse Photon Mapping for Dense Reflectometry Estimation
5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2. Photon mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3. Estimation of surface reflectance parameters considering both diffuse
and specular interreflections . . . . . . . . . . . . . . . . . . . . . .
5.4. Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.1 Preliminary experiments . . . . . . . . . . . . . . . . . . . .
5.4.2 Estimated reflectance parameters . . . . . . . . . . . . . . . .
5.4.3 Rendering results obtained using the photon mapping method
5.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vi

35
36
38
38
43
44
48
49
49
50
54
54
56
58
61
62
65
66
67
67
68
71
75
75
78
85
87

6

5.6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

87Conclusions

89

References

93

List of Publications

101

vii

List of Figures
1.1
1.2

Inverse local rendering. . . . . . . . . . . . . . . . . . . . . . . . . .
Inverse global rendering. . . . . . . . . . . . . . . . . . . . . . . . .

35

2.1 General flow diagram of surface reflectance modeling examined herein. 12
2.2 3D-digitizer and object range and texture images. . . . . . . . . . . . 13
2.3 Direction of the camera in the laser rangefinder. . . . . . . . . . . . . 14
2.4 Coordinate system of the Cyberware laser rangefinder. . . . . . . . . 14
2.5 Local surface fitting using possible windows. . . . . . . . . . . . . . 16
2.6 Multiple possible light source positions. . . . . . . . . . . . . . . . . 18
2.7 Diffuse and Specular reflection on an object surface. . . . . . . . . . 21
2.8 Detection of a self-shadow. . . . . . . . . . . . . . . . . . . . . . . . 23
2.9 Observation of only the diffuse reflection component. . . . . . . . . . 25
2.10 Observation of only the specular reflection component. . . . . . . . . 25
2.11 Four objects used in the experiment. . . . . . . . . . . . . . . . . . . 26
2.12 Measurability of both reflection components for Object A. . . . . . . 28
2.13 Measurability of both reflection components for Object B. . . . . . . 29
2.14 Measurability of both reflection components for Object C. . . . . . . 30
2.15 Measurability of both reflection components for Object D. . . . . . . 31
3.1
3.2
3.3
3.4
3.5
3.6
3.7

Flow diagram of inverse local rendering for surface reflectance modeling. 35
Point on the Object in multiple texture images. . . . . . . . . . . . . 36
A window for linearly interpolating undetermined pixels. . . . . . . . 37
Estimated reflectance parameters for Object A. . . . . . . . . . . . . 39
Estimated reflectance parameters for Object B. . . . . . . . . . . . . 40
Estimated reflectance parameters for Object C. . . . . . . . . . . . . 41
Estimated reflectance parameters for Object D. . . . . . . . . . . . . 42
ix

3.8
3.9

Rendering results for Object A with the Torrance-Sparrow model. . .
Rendering results for Object B with the Torrance-Sparrow model. . .

43
43

3.10 Rendering results for Object C with the Torrance-Sparrow model. . .
3.11 Rendering results for Object D with the Torrance-Sparrow model. . .
3.12 An object with interreflections (Object E). . . . . . . . . . . . . . . .
3.13 Position of Object E in the experiment. . . . . . . . . . . . . . . . . .
3.14 Estimated diffuse reflectance parameter in Experimental Setup 1. . . .

44
44
45
45
46

3.15 Estimated diffuse reflectance parameter in Experimental Setup 2. . . .

47

4.1

Computing the form factor between a patch and a differential area. . .

51

4.2
4.3
4.4

Hemisphere method for efficient calculation of form factor. . . . . . .
Hemicube method improving the hemisphere method. . . . . . . . . .
Flow diagram of estimating surface reflectance properties based on inverse radiosity rendering. . . . . . . . . . . . . . . . . . . . . . . . .
Calculation of radiosity. . . . . . . . . . . . . . . . . . . . . . . . . .

53
53

Comparison with previous results for Object E with diffuse surfaces. .
Comparison with previous results for Object E with glossy surfaces. .
Difference between diffuse reflectance parameters estimated with inverse radiosity and those estimated with the method in Chapter 3. . . .
Rendering results for Object A with the Torrance-Sparrow model. . .

59
60

4.5
4.6
4.7
4.8
4.9

55
56

61
63

4.10 Rendering results for Object B with the Torrance-Sparrow model. . .
4.11 Rendering results for Object C with the Torrance-Sparrow model. . .
4.12 Rendering results for Object D with the Torrance-Sparrow model. . .
4.13 Difference between rendering results based on the inverse radiosity
rendering and the inverse local rendering in Chapter 3. . . . . . . . .

63
63
63

5.1
5.2

Geometrical arrangement for photon mapping rendering. . . . . . . .
Rendering results of photon mapping. . . . . . . . . . . . . . . . . .

69
70

5.3

Flow diagram for the estimation of surface reflectance properties basedon inverse photon mapping. . . . . . . . . . . . . . . . . . . . . . . .
Detail of reflectance estimation process based on inverse photon mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Comparison of the three proposed methods for Object E with diffuse
surfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.4
5.5

x

64

72
73
76

5.6

Comparison of the three proposed methods for Object E with glossy
surfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7 Differences between real and synthetic cylindrical images for Object A.
5.8 Differences between real and synthetic cylindrical images for Object B.
5.9 Differences between real and synthetic cylindrical images for Object C.
5.10 Differences between real and synthetic cylindrical images for Object D.
5.11 Relationship between the number of iterations and difference between
real and synthetic images for Objects A  D. . . . . . . . . . . . . .
5.12 Rendering results for Object A  D under arbitrary illumination conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xi

77
80
81
82
83
84
86

List of Tables
2.1
2.2

5.1
5.2

Specifications of the Cyberware laser rangefinder. . . . . . . . . . . .
Measurability of both reflection components and the number of selected light sources for estimating the reflectance parameters of Objects A, B, C and D. . . . . . . . . . . . . . . . . . . . . . . . . . . .

14Comparison of computational costs of the three proposed methods. . .
Average and variance of differences between real and synthetic images.

78
79

xiii

32

List of Symbols












 

  





Æ




 



:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:

undetermined parameter.
undetermined parameter.
undetermined parameter.
undetermined parameter.
undetermined parameter.
undetermined parameter.
undetermined parameter.
3D position of an object surface.
3D position of an object surface.
 component of object surface point .
component of object surface point .
component of object surface point .
range value at (s,t) in a cylindrical image.
3D position at (s,t) in a cylindrical image.
gradient component of the 3D position (s,t).
gradient component of the 3D position (s,t).
gradient component of the cylindrical image r(s,t).
gradient component of the cylindrical image r(s,t).
unit vector of a surface normal.
 component of unit normal vector N.
component of unit normal vector N.
component of unit normal vector N.
range value at (s,t) on a quadratic surface.
gradient component of the quadratic surface  .
gradient component of the quadratic surface  .
xv

a
b
c
d
e
f
err(s,t)



:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:



:

 

:




:
:
:



:



:



:
:
:
:

















coefficient of the quadratic surface.
coefficient of the quadratic surface.
coefficient of the quadratic surface.
coefficient of the quadratic surface.
coefficient of the quadratic surface.
coefficient of the quadratic surface.
error function between real data and model data.
image intensity.
image intensity.
index of a light source.
index of a light source.
 coordinate of light source at .
coordinate of light source at .
coordinate of light source at .
color image obtained with light position .
color image consists of only the diffuse reflection component
with light position .
color image consists of only the specular reflection component
with light position .
color image including strong specular reflection component
with light position .
number of pixels or patches.
image intensity corresponding to the surface point .
image intensity in image  corresponding tosurface point .
number of pixels that include only the diffuse reflection component
with light source .
number of pixels that include only the specular reflection component
with light source .
threshold used in the light selection method.
diffuse reflection component of image intensity .
specular reflection component of image intensity .
image intensity in image  corresponding toxvi

:

 

:

Y
C

:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:





Ä
Î
Ê









Ä
Ê



W









:
:
:
:
:
:
:
:
:

surface point .
image intensity in image  corresponding tosurface point .
image intensity in image   corresponding tosurface point .
strength of a light source.
distance between a light source and an object surface point.
diffuse reflectance parameter at surface point .
specular reflectance parameter at surface point .
surface roughness parameter at surface point .
unit vector of a light source at surface point .
unit viewing vector at surface point .
unit half vector between  and  at surface point .
angle between L and N at surface point .
angle between V and N at surface point .
angle between V and R at surface point .
number of possible light positions.
number of selected light positions.
pixel containing only the diffuse reflection component.
pixel containing the strong specular reflection component.
pixel classified as neither  nor  .
unit light source vector at surface point  in image   .
unit half vector between  and  at surface point 
in image .
threshold angle between  and .
threshold angle between  and .
window size for linear interpolation.
radiosity of patch .
heat rate at which light is emitted from patch .
diffuse reflectance parameter of patch .
area of patch .
form factor between patch  and patch  .
direction of outgoing radiance.

Ä

Æ

Ä

Æ

Î
Î

Ê
Ê

xvii




!
¼




" 


" 
¼

"
"


#  

¼

#
#





:
:
:
:
:
:
:
:
:
:
:

direction of incoming radiance.
differential solid angle.
hemisphere of directions.
.
radiance at surface point  in direction 
¼ .
incident radiance at surface point  from direction 
emitted radiance.
reflected radiance.
BRDF.
diffuse BRDF.
specular BRDF.
threshold used in the inverse photon mapping.

xviii

Chapter 1
Introduction
Computer graphics (CG) is being increasingly used to visualize real objects and environments. Applications for entertainment, architecture, interior design, virtual reality,and digital museums often require aspects of the real world to be rendered realisticallyfrom novel viewpoints and/or under novel illumination conditions. For example, it
might be necessary to show how a room in a house would look under different lighting
conditions or how a statue would look at various times of day in a different wing ofa museum. Another example is the realistic rendering a film location under different
lighting conditions, while adding in digital props and characters, with the expectationthat the rendered results would accurately and realistically portray the desired scene.
Moreover, augmented reality (AR) technologies have been developed. Augmented reality techniques require a virtualized object to be seamlessly merged into the real world[Azu97].
There are two approaches to the problem of virtualizing real objects photo-realistically.
The first approach is image based rendering (IBR), which has often been used to reproduce real objects in CG. Generally, IBR methods require a large number of real
images in order to represent the virtualized object under arbitrary illumination conditions and from an arbitrary viewing direction. Research in image-based modelingand rendering[Che95, DYB98, DM96, LF94, MB95, LH96, SS97] has shown that thephotographics of a scene can be used together with geometry to produce realistic renderings of diffuse scenes under the original lighting conditions. However, challenges
remain with respect to making modifications to such scenes. Whether the modifications involve changing the geometry or changing the lighting, the generation of a1new rendering requires re-computation of the interaction of light with the surfaces inthe scene. Computing this interaction requires knowledge of the reflectance properties (diffuse, color, shininess, etc.) of each surface. Unfortunately, such reflectanceproperty information is not directly available from the scene geometry or from photographs. Mukaigawa et al. [MMMS01] have proposed a photometric IBR in whicha virtualized object is represented with a few real images. However, the use of a limited number of images causes a problem, whereby the appearance of the object is notfaithfully reproduced because some parts of the object are interpolated linearly.
The second approach is inverse rendering. Producing a photorealistic image requires the object geometry, reflectance properties and lighting effect in a scene to be
known. Several studies have been performed in the fields of computer vision and computer graphics in order to estimate various types of information from images in scene
or objects. For example, for the estimation of object geometry from a sparse set of images, the photometric stereo method has been developed[BH85, Hor90, LR85, Pen82,
DM03], and many approaches have been developed for estimating the lighting effectin a scene [HA93, RH01a, RH01b, RH02, SSI03]. In each of these methods, the environmental lighting condition in the scene is estimated for merging the virtual object
into the real world. Recent studies have focused on scene lighting estimation basedon spherical harmonics [RH01b, RH02], and the results of these studies have proventhat lighting condition estimation is useful for representing virtual objects in the realworld. Finally, approaches have also been proposed that can estimate object reflectanceproperties[MTY01b, MTY01a, MTY02]. The present study focuses on the estimationof object reflectance properties based on inverse rendering. This approach, which issometimes referred to as inverse reflectometry, reproduces the object shape and surface reflectance properties. If the object surface reflectance properties are estimatedsimultaneously, then the virtualized object can be rendered appropriately under virtualized illumination conditions estimated from real environments [YDMH99, SSI99,
BRG92, IS91, KC94, LL99b, LL99a, LL95, SWI97, MTY01b, MTY01a, MTY02].
Such methods employ surface reflectance models with several parameters, and theshape and color information of the object are used to estimate the reflectance parameters. The present study focuses on the estimation of object surface reflection in an
inverse rendering framework.
Generally, when light encounters an obstacle, it is either scattered or absorbed. The
2

Geometry
Reflectance
Properties

Local
Illumination
Rendering

Rendered
Image

Lighting

Geometry
Reflectance
Properties

Inverse Local
Illumination
Rendering

Rendered
Image
Lighting

Figure 1.1. Inverse local rendering.

surface of the obstacle can be made up of different materials or media. In CG, suchan effect is referred to as local surface rendering. Object surface reflection consistsof both diffuse and specular components. The diffuse reflection component is easily
observed due to the omni-directional nature of this reflection. In contrast, the specularreflection component can be observed only within a fixed range of angles with respect
to viewing position, light source, and object surface normal vector. In other words,
if the object geometry, object surface reflectance, light source direction, and viewing
direction are known, then the object surface can be photorealistically rendered using a
local illumination model. Therefore, in the case of estimating reflectance properties, if
the final rendered image, object geometry, viewing direction, and light source direction
are known, then the object reflectance properties can be obtained. This is referred toas Inverse Local Rendering(Figure 1.1).
In a number of studies [BRG92, IS91, LL95] on Inverse Local Rendering, objects
are assumed to have a uniform reflectance property over the entire surface. Reflectanceparameters are estimated using the standard least-squares method to fit a reflectance3

model to a given color image. Due to the above assumption, such methods cannot
be applied to objects that consist of several different materials and have non-uniform
reflectance properties. On the other hand, in order to treat non-uniform surface objects, some studies have used multiple images of an object under different lighting
conditions and have estimated reflectance parameters by solving simultaneous equations [KC94, LL99b, LL99a]. However, such methods still have a problem in that theresults are not stable, especially when the specular reflection component is very small.
Sato et al. [SWI97] developed a method by which to estimate non-uniform reflectanceproperties. In their method, an object was placed on a robot arm, and was then measured with a CCD camera and a rangefinder from a large number of viewpoints byrotating the robot arm. Reflectance parameters were stably acquired by decomposing
the surface reflection into two components based on the singular value decomposition(SVD) technique. Although the method can be applied to objects with non-uniform
reflectance properties, the shape of object should be limited because it is difficult toobserve the specular reflection component over the entire surface. This is because thelighting conditions for each pose of the object with respect to the camera cannot bechanged in this method.
The studies mentioned above focus on the local illumination framework which
deals with only the direct illumination. In the real world, surfaces usually exhibit
mutual illumination. Thus, the light that any particular surface receives may originate
from not only the light sources, but also from the rest of the environment, throughindirect illumination. For example, a wall in a room receives lighting effects fromother walls that are illuminated by the light sources. Therefore, when object surface
reflectance parameters are estimated, it is important to consider interreflections. As aresult, the incident radiance of an observed surface is a complex function of the lightsources, the geometry of the scene, and the undetermined reflectance properties of allof the surfaces in the scene. Therefore, the incident radiances of the surfaces mustbe estimated. This allows the reflectance properties of the surfaces in the scene to be
estimated via an iterative optimization procedure, which allows the incident radiances
to be re-estimated. This is referred to as Inverse Global Rendering(Figure 1.2).
In order to consider interreflections, it is important to employ a global illumination
model. The physically-based simulation of all light scattering in a synthetic model isreferred to as global illumination. The goal of global illumination is to simulate all of4

Geometry
Reflectance
Properties

GlobalIllumination
Rendering

Rendered
Image

Lighting

Geometry
Reflectance
Properties

Inverse Global
Illumination
Rendering

Rendered
Image
Lighting

Figure 1.2. Inverse global rendering.

the reflections of light in a model and enable an accurate prediction of the intensity of
the light at any point in the model. The input to a global illumination simulation is a
description of the geometry and the materials, as well as the light sources. The global
illumination algorithm is used to compute how light leaving the light sources interacts
with the scene. Several global illumination algorithms have been developed[Kaj86,
ICG86, WCG87, Jen01], most of which are based on two major techniques: Point sampling (ray tracing) Finite elements (radiosity)
In addition, hybrid techniques exist that combine radiosity and ray tracing.
The ray tracing method is a point-sampling technique that traces infinitesimal beamsof light through a model. This approach can only handle mirror reflections/refractions
and direct illumination. Important effects, such as depth of field, motion blur, caustics,
indirect illumination, and glossy reflection cannot be computed. In order to simulate these effects, ray tracing has been extended with Monte Carlo methods[Coo86,
5CPC84, Kaj86], in which rays are distributed stochastically to account for all of thelight paths. However, Monte Carlo ray-tracing has a problem in that rendered images include some noise, and eliminating this noise requires a large number of sample
rays[Jen95, LW95, Shi91, VG95, VG97].
On the other hand, finite element radiosity techniques are an alternative to raytracing methods, in which the equilibrium of the light exchange between surfaces ina model is computed. This is done by subdividing the model into small patches thatcan form a basis for the final light distribution. The lighting distribution is found bysolving a set of linear equations for the light exchange between all of the patches. thistechnique can represent indirect illumination (interreflections). Recently, radiosity isextended to handle more complex reflection models[ICG86, WCG87, SAW91, SCH93,
CSSD96, SSS00], but simple curved specular surfaces are still not handled properly.
The photon mapping method has also been recently developed. Photon mappingchanges the representation of the illumination. Instead of tightly coupling lighting
information with the geometry, the information is stored in a separate independent
data structure, the photon map[Jen01]. The photon map is constructed from photons
that are emitted from the light sources and are traced through the model. The photon
map contains information about all photon hits, and this information can be used toefficiently render the model in a manner similar to that in which radiosity is used in
hybrid techniques. The decoupling of the photon map from the geometry is a significant advantage that not only simplifies the representation but also makes it possible
to use the structure to represent lighting in very complex models. The combination ofphoton mapping and a Monte Carlo ray-tracing algorithm result in an algorithm that is
as general as pure Monte Carlo ray tracing, but that is significantly more efficient.
For the reflectance estimation problem, Fournier et al. [FGR93] originally developed a method for estimating surface reflectance properties using the radiosity method
in an indoor scene. This method assumes uniform reflectance on an object surface
and determines only the diffuse reflectance parameter. There have been a number ofattempts to estimate both the diffuse and specular reflectance properties of a room
[BG01, LDR00]. In these studies, however, all objects are also assumed to have uniform reflectance properties in the segmented region. Therefore, these algorithm cannot
be applied to an objects that have non-uniform surface reflectance properties. Yu et al.
[YDMH99] estimated the surface reflectance properties of a room from color and ge6

ometry data considering both diffuse and specular interreflections based on inverse
global illumination rendering. They employed a hybrid rendering method that combines radiosity and the Monte Carlo ray tracing method. Boivin et al. [BG01] havealso attempted to estimate the surface reflectance properties considering diffuse interreflections. These methods assume that the surface of interest has uniform reflectanceproperties. Therefore their algorithms cannot be applied to a non-uniform surface reflectance object.
The present study focuses on the estimation of non-uniform surface reflectanceproperties based on the inverse rendering framework, as mentioned previously. First,
a method for estimating local surface reflectance properties is proposed. For this purpose, optimum light positions are selected in order to observe both diffuse and specularreflection on the object surfaces. By this process, both diffuse and specular components at each surface point can be decomposed. Then, the object surface reflectanceproperties are densely estimated based on Inverse Local Rendering. Specifically, theTorrance-Sparrow model, which is the most popular local object surface reflectancemodel and can represent physical phenomena, is employed. The proposed method
can estimate the object surface reflectance parameters at each point on the object
with results that are comparable to those of conventional methods[MTY01b, MTY01a,
MTY02].
Inverse Local Rendering can estimate the object surface reflectance properties based
only on the direct illumination effect. However, in the real world, objects have interreflections. Therefore, the reflectance properties must be estimated considering interreflections on the surface. In order to solve this problem, two estimation methods areproposed. One is based on the radiosity rendering method and the Torrance-Sparrowreflectance model, and the other is based on the photon mapping rendering method.
The former method can estimate diffuse reflectance parameter by calculating the inverse radiosity algorithm and the specular and surface roughness parameters by calculating the inverse local reflectance model. The advantage of this method is that
diffuse interreflections can be considered[MY02b, MY02a, MYT03]. In the inverse
radiosity method, the influence of the specular interreflections remains. As a result,
this method cannot be applied to objects that have complex geometry and specular reflection. Moreover, although the diffuse reflectance parameter can be estimated basedon global illumination rendering, the specular reflectance and the surface roughness
7

parameter can only be estimated by local illumination rendering. Therefore, in orderto address the shortcomings of the inverse radiosity estimation method, the photon
mapping rendering method, which can also estimate diffuse reflectance, specular reflectance and the surface roughness parameters, was proposed. In this method, sinceboth diffuse and specular interreflections can be considered, the inverse photon mapping method is a more efficient estimation method compared to the conventional methods. In addition, because the photon mapping rendering method has recently been implemented on graphics hardware (GPU)[PDC  03, PBMH02], real-time estimation of
the object reflectance parameters is desired.
Finally, in experiments, the three proposed methods are qualitatively and quantitatively evaluated with reflectance parameter estimation and virtual relighting. These
three methods are compared, and Inverse Global Rendering is also shown to be useful for estimating object surface reflectance properties. The radiosity method and thephoton mapping method can usually be applied to a closed environment. However,
the inverse radiosity and the inverse photon mapping are experimentally verified to be
applicable to an open environment, in which the influence of environmental light isnegligible.
The remainder of this thesis is structured as follows. Chapter 2 proposes a method
for effective observation of surface reflections. As described above, the object surface
reflection consists of two components: the diffuse reflection component, which is reflected by the light source omni-directionally, and the specular reflection component.
The specular reflection component can be observed only at a fixed angle according tothe object geometry, the light source direction, and the viewing direction. Due to theirnature, the specular reflectance properties of an object are difficult to densely estimate,
and thus usually require the assumption of uniform properties. In Chapter 2, the problem of how to observe both diffuse and specular reflection components at each surface
point is described.
The latter part of this thesis discusses how to estimate the object surface reflectanceproperties. A previously obtained image, which consists of both diffuse and specularreflection components densely, is used for estimating surface reflectance properties.
The object geometry is also obtained using a laser rangefinder. Basically, if the object geometry, object texture, light source direction and viewing direction are known,
then the Inverse Rendering method can be used to find unknown surface reflectance8parameters. For this purpose, three reflectance estimation methods are proposed.
Chapter 3 proposes a method for automatically estimating object surface reflectanceproperties based on Inverse Local Rendering. The application of inverse rendering has
recently become widespread. In CG, a photorealistic image is rendered using a surface
reflectance model that can represent physical phenomena in the real world. In particular, color determination of object surface is calculated by local illumination model,
which can represent the influence of direct illumination. In Chapter 3, local surface
reflectance properties are estimated using the Torrance-Sparrow model as a local illumination model without the assumption of uniform surface reflectance properties. The
effectiveness of this method is verified experimentally using a number of objects with
non-uniform surface reflectance properties.
Chapters 4 and 5 focus on Inverse Global Rendering. Inverse local rendering candeal with only the influence of direct illumination. Due to this limitation, if an object
has interreflections, then the estimated reflectance parameters have some errors andrendering the object photorealistically is difficult. In order to solve this problem, theglobal illumination model, which can represent interreflections in the real world, isemployed. In Chapter 4, a radiosity rendering method is employed that can represent
diffuse interreflections. Using this method, inverse radiosity rendering can be used
to estimate non-uniform surface reflectance properties with interreflections. However,radiosity rendering can represent only diffuse interreflections. Due to their nature,
specular interreflections cause some errors in estimated results. In Chapter 5, in orderto solve the problem of the inverse radiosity rendering method, a photon mapping
rendering method is employed. This rendering model can efficiently treat all influencesof lighting effects in the real world. Finally, experimental results are discussed using
uniform or non-uniform objects that have both diffuse and specular interreflections.
Finally, Chapter 6 presents the conclusions of primary importance obtained in thisstudy and discusses possible subjects for future research.

9Chapter 2
Dense Observation of Reflection
Components
2.1. Introduction
In this chapter, the optimum light positions are selected for observing both diffuse andspecular reflection components over the entire surface of an object. Because the diffuse reflection component is observed omni-directionally on the surface, it is easy toobserve. On the other hand, the specular reflection component can be observed on onlya certain fixed area on the object surface. Therefore, several light sources located at
different positions are needed in order to observe the specular reflection component.
In other words, if the specular reflection component is not observed, the specular reflectance parameters cannot be estimated. Furthermore, the greater the number of lightsources used, the more images should be acquired. This results in a large computational cost. Therefore, it is important to determine the optimum light position so thatboth diffuse and specular reflection components are observed densely with the minimum cost.
Figure 2.1 shows a flow diagram of the light selection (A, B, C) and object surface reflectance estimation (D) processes. This chapter focuses on the light selection
process. A laser rangefinder was used to obtain range and surface color images, and optimum light positions were then selected based on the object range image. The surface
reflectance estimation process (D) is described in the next chapter. The light selection
process consists of two processes: measurement of an object (A, C) and selection of11

A. Measurement
Range data
B. Selection of optimum positions of light sources
Selected light position 1… Selected light position m

C. Measurement
Surface texture image 1… Surface texture image m

D. Object surface reflectance estimation process
Object surface reflectance parameters

Figure 2.1. General flow diagram of surface reflectance modeling examined herein.

light sources (B). The following presents an overview of the light selection process.
1. Measurement of 3D geometry and surface texture
The object shape is acquired using a laser rangefinder in a dark room. Multiple surface texture images are also obtained under different light sources, thepositions of which are selected in process (B).
2. Preprocessing
Generally, a range image acquired by laser rangefinder contains some noises,
including quantization errors. Moreover, a normal vector cannot be correctly
calculated in the vicinity of discontinuities. Therefore, noises are removed andquadratic surfaces are fit locally to the range image in order to acquire the normal
vector correctly.
3. Selection of light source
According to the object shape and the position of the camera, multiple light12

Figure 2.2. 3D-digitizer and object range and texture images.
positions are selected from among a number of possible light source positions inorder to measure both diffuse and specular reflection components densely on theobject surface.
The following sections describe the optimum light position selection method in detail.

2.2. 3D measurement of an object
A laser rangefinder (Cyberware 3030RGB) with point light sources at known positions
and a camera were used to acquire range and surface color images of an object locatedin a dark room, as shown in Figure 2.2. This system can simultaneously obtain registered range and surface color texture images by rotating the rangefinder and the camera
around the object. Figure 2.3 shows a the device as viewed from above. The camera
is located at $  and a texture image is acquired through mirrors located at $  and$ . Thus, the camera is assumed to be virtually located at $ . The camera is directed
toward the center of rotation.
The specifications of the rangefinder are listed in Table 2.2. The coordinates systemof the texture and the range image are also shown in Figure 2.4. Here, % is the distancefrom the center of rotation to the object surface. The vertical axis  indicates the object
height, and the horizontal axis  indicates the rotation angle from the original position.
13

Center of rotation

Laser rangefinder
X1
X2
CameraX3 Virtual camera
X4

Figure 2.3. Direction of the camera in the laser rangefinder.

Table 2.1. Specifications of the Cyberware laser rangefinder.
Image size (WH)
512450
Vertical resolution (mm)0.56
Horizontal resolution (degrees)
0.7
Color resolution
8-bit
Range resolution
8-bit

R

h

ObjectCylindrical range image

θFigure 2.4. Coordinate system of the Cyberware laser rangefinder.

14

2.3. Preprocessing
Generally, the noise and quantization errors are included in the range image acquired
using the laser rangefinder. Another problem is that the surface normal is not calculated
accurately around the discontinuity in the range image. Therefore, an adaptive local
quadratic surface fitting [YL89] is employed as a preprocessing.
First, a 55 median filter is applied to the range image to remove the noise (outliers). Second, the quadratic surface is locally fitted with the range image. The range
image obtained using the Cyberware laser rangefinder is expressed in cylindrical coordinates, as mentioned earlier. The 3D position      at each point of the range
image is given as:
 








  Ë      



(2.1)
(2.2)

where , , and  are the distance from the center of rotation, the angle of rotation, andthe height along the rotation axis, respectively.
The unit normal vector is given as follows:Æ  ËË  ËË












(2.3)



 



  

 

 



  

 

 



(2.4)

where  and  are gradient components of the range image  . This gradient isanalytically computed using the following local quadratic surface fit:   &  '  (  !  )  #(2.5)

  &  (  !

(2.6)

  '  (  )*(2.7)

Coefficients &  # are determined by minimizing the following equation using therange data   and Equation (2.5):
)

 








                   
15

(2.8)

possible windows(s,t)

Figure 2.5. Local surface fitting using possible windows.

where  and  are local coordinates in a    window. In the present approach, selected
local quadratic surface fit is achieved using the Yokoya-Levine operator [YL89]. Usinga    window, the best window is selected from among 25 possible windows that
include the point   in order to estimate the coefficients &  # at   (See Figure2.5). The best window provides the minimum fitting error in Equation (2.8).

16

2.4. Selection of light source positions based on the reflectance model
Here, the problem of determining the light source positions for effectively observing
both the diffuse and specular reflection components on an object surface is addressed.
In the present experimental setup, multiple positions of a light source are determinedamong 60 possible positions prepared around the laser rangefinder. These positions arearranged two-dimensionally at 5-cm intervals, as shown in Figure 2.6. After optimum
light positions are selected, a single light source is attached at the selected positions inturn so that brightness calibration among multiple lights is not needed. Note that thecamera position is calibrated in advance by measuring a calibration box of known size.
The position of a light source is also calibrated based on the distance from the centerof rotation in global coordinates. In addition, the influence of environmental light canbe ignored because the object is measured in a dark room.
Let  be a color image that is to be obtained with a possible light position    and that consists of  pixels  , where  is a color intensity
corresponding to the surface point . Also, let  be the number of pixels that include
only the diffuse reflection component in   , and let  be the number of pixels that
include the strong specular reflection in  .
First, the following conditions are examined for each pixel in the object surface
texture under each light position : Measurability of light reflection
 Measurability of only diffuse reflection
 Measurability of strong specular reflection  and  
Second, light positions  and  that satisfy    & &
 are selected. In the next light source position selection, a position that satisfies the same conditions is selected from among the remaining possible
positions. Then,  light positions are selected to densely estimate reflectance parameters. The selection of light positions is repeated until almost all of the pixels havebeen observed once for the diffuse reflection component only and twice for the strong
specular reflection component.

17

Laser rangefinder

Possible light position
Figure 2.6. Multiple possible light source positions.

Next, the threshold , i.e., the ratio of the measurability of the diffuse reflection component to that of the strong specular reflection component, is introduced asa means of determining when to stop the process of selecting light positions. With
respect to determining the threshold  , the ratio of measurability of the specular reflection is judged at all possible positions. This ratio indicates the limit for measuring
the specular reflection component. Using this threshold, the light selection process
can be terminated in the case that the ratio of measurability of the specular reflection
component cannot be increased further, even if the number of light source positions isincreased. Thus, the reflectance parameters can be estimated efficiently for almost theentire object surface using a limited number of texture images. In the following, after
presenting the surface reflection model used in the present study, the three measurability conditions mentioned above are described in detail.

2.4.1 Torrance-Sparrow model
Selecting the optimum light positions from among the possible light positions requires
judgment as to whether a pixel in the image taken by a laser rangefinder has each reflection component. This requires a surface reflection model that represents the propertiesof both diffuse and specular reflection components. The reflection model describes
the direction of reflection from a light source based on object geometry, viewing direction and light source direction. If these parameters are known, each pixel in theimage can be judged as to whether it has both reflection components. In this chap18

ter, the Torrance-Sparrow model [TS67] is used to represent local surface reflections
physically. The Torrance-Sparrow model is given as:

















+
,




 

+




)








(2.9)

   
,





)  


 


(2.10)(2.11)(2.12)Æ Ä  Æ Î  Æ Ê Ê  ÎÎ  ÄÄ(2.13)(2.14)(2.15)(2.16)

where  represents an observed intensity corresponding to the surface point ,   and denote the diffuse and specular reflection components, respectively, , is an attenuation coefficient related to the distance between a point light source and an object
surface point, and + represents the strength of a light source. In addition,   , and  are the diffuse reflectance parameter, specular reflectance parameter, and surface roughness parameter, respectively, which is the standard deviation of a Gaussian
distribution, respectively,  is the angle between light source vector  and surface
normal vector ,  is the angle between viewing vector  and surface normal
vector  , and  is the angle between surface normal vector  and half vector  .
Note that half vector  is the vector located halfway between light vector  andviewing vector  , and is given by Equation (2.16). All vectors are unit vectors. Figure 2.7 illustrates the geometry for this model. The angle   is used to judge whether
the specular reflection occurs.
In the case of using a color image,  ,  ,  ,  and  consist of RGB channels, and the model is applied to each channel independently. In order to estimate
reflectance parameters  ,  and  using this model, it is necessary to obtain theother parameters, i.e., , , , + , ,  and  , at each point on the object surface.
Dense and independent estimation for non-uniform surface reflectance parameters
requires observation of each surface point  under at least three different lighting
conditions: one lighting condition for determining the unknown parameter   , and

Æ

Æ

Î

Î
Æ

Ê

Ä Î

Æ

19

Ä

Ä

Ê

the other two lighting conditions for acquiring the remaining two unknown parameters  and  . The selection of the optimum positions of the light source in Figure 2.1(B) is repeated until almost all of the pixels satisfy the three different lighting
conditions[MTY01b]. As a result of this process, a certain number of light positions,say  (as described previously), are selected in order to densely observe both diffuse
and specular reflection components.
 andA texture image is obtained with a selected light position     , where  is the color intensity of surface point .
consists of  pixels 20

Normalvector N
Light source
vector L

θdRange for observing
the diffuse reflection

Object surface(a) Diffuse reflection

Normalvector N

Halfvector R

θrViewing
vector V

Light source
vector Lθdθv

Range for observing
the specular reflection

Object surface(b) Specular reflection
Figure 2.7. Diffuse and Specular reflection on an object surface.

21

2.4.2 Measurability of light reflection
In order to measure the light reflection at a specific point  on the object surface, thesurface point must be observable from the camera position and the light must illuminate
the surface point. Thus, the positional relationship among the camera, the point andthe light source must satisfy the following conditions.

Î Æ -  Ä Æ - Î Ä(2.17)

Æ

where  , , and  are the viewing direction, the light source direction, and thesurface normal corresponding to the surface point , respectively. Note that the viewing direction  and the surface normal  are independent of the light source position
.
Even when the above equations are both satisfied, the possibility exists that ashadow is cast on the pixel. Figure 2.8 illustrates such a case. In this case, the pixel
must not be used for estimating reflectance parameters. Whether a surface point  iscovered by a shadow cast by light source  can be judged as follows:Let      be the coordinates of the point on the object surface, and let    be the coordinates of the possible light position . Then, there exists abounding box that is surrounded by the maximum and minimum values on each ofthe , , and axes of positions  and , as shown in Figure 2.8. Here, all polygonsthat make up the object surface are assumed to be small enough to be covered by thebounding box. Then, determining whether or not a shadow is cast over requires an
examination to determine whether any polygon that contains a vertex inside the bounding box intersects with the line segment that connects and . Therefore, for all the
points inside of the bounding box, the following test is performed [FvDFH93].
Let  be a point that is found to be included in the bounding box, and let     
be the coordinates of this point. If  lies on a surface polygon of which the normal
vector is    , then the polygon and the line segment that connects  and are
given, respectively, by the following equations:

Î

Æ




                   
 



 
    *    

(2.18)The intersection of the plane and the light segment, i.e., the point , can be obtained22pLight source

Bounding box
uo

Shadow area

Object surface
Figure 2.8. Detection of a self-shadow.
by solving Equation (2.18) for  , where  is defined as follows:Í                                

Æ Í

Æ È

(2.19)Èwhere  is the vector from the point  to the surface point , and  is the vector
from the light position  to the surface point . If  satisfies    and the pointlies on the polygon that includes , the line segment of Equation (2.18) is judged tocross the object.

23

2.4.3 Measurability of diffuse reflection only
When the pixel at surface point  exhibits only diffuse reflection, the half vector
satisfies the following equation:Ê Æ Ê -  (2.20)
where   is the threshold angle between Æ and Ê. Equation (2.20) implies that  ( 









only the diffuse reflection component is observed if   is greater than   as illustrated
in Figure 4.13. When this condition stands and the pixel is not in a shadow, then the
pixel is judged to have diffuse reflection only and is counted in   .

2.4.4 Measurability of strong specular reflection
When the pixel at surface point  exhibits strong specular reflection, the half vector
 satisfies the following equation:ÊÆ Ê  (2.21)
where   is the threshold angle between Æ and Ê. Equation (2.21) implies that  ( 









both the diffuse and specular reflection components are observed if   is smaller than , as illustrated in Figure 2.10. The above condition is based on the fact that specularreflection is observed strongly in a limited viewing angle range. When this condition
stands and the pixel is not in a shadow, then the pixel is judged to have strong specularreflection and is counted in  .

24

L
N
R

V

θr

Range for observing
both reflection
componentsθ th1
Object surface

Range for observing
only the diffuse
reflection component

Figure 2.9. Observation of only the diffuse reflection component.

V

L

R

θth2

N

θr

Range for observing
the strong specularreflection component

Object surface

Figure 2.10. Observation of only the specular reflection component.

25

(b) Object B(a) Object A(c) Object C(d) Object D

Figure 2.11. Four objects used in the experiment.

2.5. Experimental results
Next, the efficient selection of optimum light positions from a dense set of possible
light positions is demonstrated experimentally. The actual measurability of both diffuse and specular reflection components on test objects are shown. Four objects areused in these experiments (Figure 2.11). Objects A and B have non-uniform reflectanceproperties with respect to both diffuse and specular reflection and have comparatively
complicated geometries. These objects are mainly used to verify that the proposed
methods can estimate n object with non-uniform reflectance properties (Chapter 3)
and can eliminate the influence of interreflections (Chapter 4). Objects C and D have
non-uniform diffuse reflectance and uniform specular reflectance properties. These
objects are also mainly used to verify that the proposed methods can eliminate both
diffuse and specular interreflections (Chapter 5). According to Ward et al.[War92], the26

strong specular component can be observed within 20 degrees around the half vector. Therefore, the thresholds    ,    , and    are fixed. In
addition, a PC (CPU: 3.06 GHz, Memory: 1 GB) is used in all of the experiments.
Figures 2.12  2.15 illustrate the measurability of both reflection components withrespect to the number of light sources for Objects A  D. The vertical axis indicates themeasurability of the diffuse or specular reflection component, and the horizontal axis
indicates the number of selected light sources in the proposed light selection algorithm.
Note that the horizontal axis only extends to 20 because these graphs do not changeeven if all 60 possible positions are used. In each of these figures, (a) shows the relationship between the number of selected light positions and the ratio of measurabilityof the diffuse reflection component. For example, in Figure 2.12(a), when the number of selected light source positions is five, the ratio of measurability of the diffuse
reflection component is . Moreover, (b) in each of these figure shows the relationship between the number of selected light positions and the ratio of measurabilityof the specular reflection component. For example, even when Object A is measured at
all possible positions, the ratio of measurability of the specular reflection componentis *. In the proposed method, the ratio of measurability of the specular reflection component is * with 12 automatically selected light source positions. The
measurability of both reflection components and the number of selected light sources
for estimating the reflectance parameters of Objects A, B, C and D are summarized inTable 2.2. The number in brackets indicates the number of selected light source positions required to estimate the diffuse reflectance parameter. This table also compares
the proposed light selection method and the case of using all possible positions withrespect to the measurability of the specular reflection component. The specular reflection component is shown to be observed in all objects by efficiently using a limited
number of light source positions compared to using all 60 possible positions.

Ê

27

Ratio of measurability ofdiffuse reflectance parameter

100
90
80
70
60
50
40
30
20
10
0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Number of selected light source positions

Ratio of measurability of
specular reflectance andsurface roughness
parameters(a) Possibility of estimating the diffuse reflectance parameter
100
90
80
70
60
50
40
30
20
10
0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Number of selected light source positions(b) Possibility of estimating the specular reflectance and the surface
roughness parameters
Figure 2.12. Measurability of both reflection components for Object A.

28

Ratio of measurability ofdiffuse reflectance parameter

100
90
80
70
60
50
40
30
20
10
0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Number of selected light source positions

Ratio of measurability of
specular reflectance andsurface roughness
parameters(a) Possibility of estimating the diffuse reflectance parameter
100
90
80
70
60
50
40
30
20
10
0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Number of selected light source positions(b) Possibility of estimating the specular reflectance and the surface
roughness parameters
Figure 2.13. Measurability of both reflection components for Object B.

29

Ratio of measurability ofdiffuse reflectance parameter

100
90
80
70
60
50
40
30
20
10
0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Number of selected light source positions

Ratio of measurability of
specular reflectance andsurface roughness
parameters(a) Possibility of estimating the diffuse reflectance parameter
100
90
80
70
60
50
40
30
20
10
0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Number of selected light source positions(b) Possibility of estimating the specular reflectance and the surface
roughness parameters
Figure 2.14. Measurability of both reflection components for Object C.

30

Ratio of measurability ofdiffuse reflectance parameter

100
90
80
70
60
50
40
30
20
10
0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Number of selected light source positions

Ratio of measurability of
specular reflectance andsurface roughness
parameters(a) Possibility of estimating the diffuse reflectance parameter
100
90
80
70
60
50
40
30
20
10
0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Number of selected light source positions(b) Possibility of estimating the specular reflectance and the surface
roughness parameters
Figure 2.15. Measurability of both reflection components for Object D.

31

Table 2.2. Measurability of both reflection components and the number of selected
light sources for estimating the reflectance parameters of Objects A, B, C and D.
Measurement object
Number of selected light positions
Diffuse reflection component
Specular reflection component(present method)
Specular reflection component(with 60 light positions)Object A
12(5)100.0%

Object B
14(4)100.0%

Object C10(3)100.0%

Object D9(4)100%

83.46%

91.02%

89.17%

91.72%

83.52%

93.56%

91.27%

94.90%

2.6. Conclusions
In this chapter, an optimum light position selection method for densely observing both
diffuse and specular reflection components on an object surface was proposed. In
the present approach, multiple light source positions around the laser rangefinder areautomatically selected based on the object geometry, so that diffuse and specular reflection components are observed densely. Since both reflection components must be
observed at each object point for non-uniform surface reflectance estimation, the proposed method can be used for this purpose. Experiments have shown that the proposed
method is useful for observing both reflection components densely. Dense observationof both diffuse and specular reflection components was verified for the four objects
used in the experiments. However, specular reflection components can not be observed
over the entire surface of an object, through either the proposed method or using all
possible light positions. In order to address this problem, in the future, the proposed
method will be improved by employing a light planning operation, such as those described in [WF91, LLSS03]. The following chapter describes the estimation of object
surface reflectance properties based on the proposed optimum light selection method.

32

Chapter 3
Inverse Local Rendering for Dense
Reflectometry Estimation
3.1. Introduction
In this chapter, the estimation of surface reflectance properties is described based on
the registered range and color images taken by a laser rangefinder with selected light
positions in the previous chapter. In the previous chapter, color images are obtainedthat include both diffuse and specular reflection components densely on the object
surface. The surface reflectance properties are estimated using these color images
and the object geometry, and these properties are specifically fit to the local renderingmodel. The Torrance-Sparrow model [TS67] described in Chapter 2 is used as the
local rendering model. In other words, inverse local rendering is performed for thedata obtained in previous chapter. Recall that the Torrance-Sparrow model is given as:




+
,




 




)








(3.1)

where  denotes the surface point,  represents an observed intensity, , is an attenuation coefficient concerning the distance between a point light source and an object
surface point, and + represents the strength of a light source. In addition,   , and  are the diffuse reflectance, the specular reflectance and the surface roughness
parameters, respectively, and  is the angle between a light source vector and a surface normal vector,  is the angle between a viewing vector and a surface normal
33

vector, and  is the angle between a surface normal vector and a half vector. Notethat the half vector is located halfway between the light vector and the viewing vector.
Here,  is used to judge whether or not specular reflection occurs. The following
section describes how to estimate the object surface reflectance properties densely using the Torrance-Sparrow model. Note that the Torrance-Sparrow model supports both
non-metal and heterogeneous dielectric materials, and cannot be used to estimate theobject surface reflectance properties for metallic materials. In addition, although theTorrance-Sparrow model supports the Fresnel component, the Fresnel component isignored because the object is assumed to be opaque. Moreover, mirror-finish material
is not supported. Thus, the proposed estimation method using the Torrance-Sparrowmodel cannot deal with metallic materials, opaque materials, or mirror-finish materials.

3.2. Estimation of surface reflectance parameters
After the positions of the light sources are determined as described in Chapter 2, multiple color images are taken under different lighting conditions. Figure 3.1 shows a flow
diagram of surface reflectance modeling. The proposed process consists of the fourparts in Figure 2.1: measurement of an object and preprocessing (A, C), selection of alight source (B) (as described in Chapter 2), and estimation of reflectance parameters(D). In the estimation process (D), non-uniform reflectance parameters are estimatedby substituting the object shape and textures under selected illumination conditions
into the surface color reflection model. Throughout this chapter, it is assumed that
there are no interreflections on the object surface.
Let  be a set of pixels that include only the diffuse reflection component with , where  denotes thepossible light position , as well as  pixels 
color intensity corresponding to the surface point . If it is judged that   does notconsist of only the diffuse reflection component, then    . Let   be a setof pixels that include the specular reflection component strongly, as well as  pixels
   , where   denotes the color intensity on the object surface point .
If it is judged that    does not include the strong specular reflection component, then   . In addition, denotes a pixel containing only the diffuse reflection component, and  denotes a pixel that contains a strong specular reflection component.
Moreover,  denotes a pixel that is classified as being neither  nor  . Each
34

Range datam surface texture images(a) Estimation of diffuse reflectance parameter

Diffuse reflectance parameter(b) Estimation of specular reflectance andsurface roughness parameters

Specular reflectance and the surface roughness parameter

Figure 3.1. Flow diagram of inverse local rendering for surface reflectancemodeling.

pixel is classified as belonging to either ,  or  .

3.2.1 Estimation of diffuse reflectance parameter
The estimation process is performed at each point on the object surface. Pixels in
multiple images corresponding to the object surface are used, as shown in Figure 3.2.
The diffuse reflectance parameter  at surface point  is estimated by solving thefollowing equation from Equation (2.11).








+


,
,  
+



(3.2)
(3.3)



where  is the value of the pixel at surface point  in the image  , and
is calculated from surface normal vector  and light source vector . In orderto obtain the most reliable estimation, the pixel having the smallest angle   that isgreater than  is selected.

Æ

35

Ä

Figure 3.2. Point on the Object in multiple texture images.

3.2.2 Estimation of specular reflectance and surface roughness parameters
The specular reflectance parameter  and the surface roughness parameter  at surface point  are also estimated by solving Equation (2.12) with , ,  and thevalue of the specular reflection component that is extracted from pixels    and  
in images   and  . In order to obtain the most reliable estimation, either the
pixels having the smallest angle  , or those having the second smallest angle  thatare smaller than , are selected.

Æ Ä ÎFirst, the diffuse reflection component is computed with the diffuse reflectanceparameter  , estimated above, and Equation (2.11). Next, the specular reflection
component is extracted by subtracting the diffuse reflection component from Equation(2.10).
  

   

   

(3.4)

Finally, the specular reflectance and surface roughness parameters are obtained by36

W

Subject pixel

W

Figure 3.3. A window for linearly interpolating undetermined pixels.

solving Equation (2.12).

 



 











 )
.  ). (3.5)

Equation (2.12) cannot be solved for pixels at which strong specular reflection isobserved less than twice. In such cases, the specular reflectance and surface roughness
parameters are obtained using a linear-interpolation method within a /  / window(Figure 3.3).

37

3.3. Experimental results
In order to verify the proposed method, experiments were carried out to densely estimate the reflectance parameters of Objects A  D, shown in Figure 2.11. Note thatwith respect to the window size used in interpolation method, /   is fixed.

3.3.1 Estimated reflectance parameters
Estimated parameters for the objects are shown in Figures 3.43.7, where the parameter values are represented by color or intensity in cylindrical coordinates. The diffuse
reflectance parameter estimated over the object surface is shown in (a) in each figure.
The specular reflectance and surface roughness parameters are shown in (b) and (c),respectively, in each figure. Note that in each figure (c) is illustrated in gray scale, inwhich the largest value is represented as white. This image indicates that the smaller
the value, the smoother the object surface. In addition, (d) in each figure illustrates
the ratio of pixels at which the specular reflectance and surface roughness parameters
can be computed. The black area indicates the area in which neither parameter can bedirectly estimated.
In Figure 3.4, the specular reflectance and surface roughness parameters of thebeak and leg of the object clearly differ from those of the other areas. Actually, thebeak and legs are highly reflective, as can be seen in Figure 2.11(a). In Figure 3.5, thereflectance parameters of Object B, having a more complex geometry, are estimated.
This figure shows that some parts have the diffuse reflection component and otherparts have the specular reflection component, as is the case for Object A. These results
verify that non-uniform surface reflectance properties can be observed efficiently using
the proposed light selection method.
Objects C and D also have specular reflection components. These objects actually have uniform specular reflectance and surface roughness parameters, as can beobserved in Figure 2.11(c) and (d). These observations can be confirmed from Figures
3.6 (b) and (c) and 3.7 (b) and (c).
These results also show that uniform of non-uniform specular reflectance and surface roughness parameters are estimated without interpolation for more than nearly of the surface.

38

(a) Diffuse reflectance parameter   (b) Specular reflectance parameter   (c) Surface roughness parameter   (d) Observation area for specular reflection
Figure 3.4. Estimated reflectance parameters for Object A.
39

(a) Diffuse reflectance parameter   (b) Specular reflectance parameter   (c) Surface roughness parameter   (d) Observation area for specular reflection
Figure 3.5. Estimated reflectance parameters for Object B.

40

(a) Diffuse reflectance parameter   (b) Specular reflectance parameter   (c) Surface roughness parameter   (d) Observation area for specular reflection
Figure 3.6. Estimated reflectance parameters for Object C.

41

(a) Diffuse reflectance parameter   (b) Specular reflectance parameter   (c) Surface roughness parameter   (d) Observation area for specular reflection
Figure 3.7. Estimated reflectance parameters for Object D.

42

Figure 3.8. Rendering results for Object A with the Torrance-Sparrow model.

Figure 3.9. Rendering results for Object B with the Torrance-Sparrow model.

3.3.2 Rendering results with the Torrance-Sparrow model
Finally, the virtual re-lighting of virtualized objects is demonstrated. Figures 3.8 
3.11 show the rendering of virtualized Objects A, B, C and D under a virtual lighting
condition. A virtual light source is fixed at (0.0, 0.0, 300.0), and the origin of thecoordinate system is located at the center of the object. Rotation is around the verticalaxis of each object. Figure 3.8  3.11 contains six views around the virtualized objects.
Each result shows the potential usefulness of the proposed method, although somespike noises are present due to sensor error in the range data. These results qualitatively
show the effectiveness of the proposed surface reflectance modeling method.

43

Figure 3.10. Rendering results for Object C with the Torrance-Sparrow model.

Figure 3.11. Rendering results for Object D with the Torrance-Sparrow model.

3.4. Discussion
In this chapter, non-uniform object surface reflectance properties were densely estimated using a local illumination model (Torrance-Sparrow model). As described in
Chapter 2, because the local illumination model considers only the direct illumination
effect on the object surface, errors may exist if interreflections occur. In this section,
the degree to which errors due to the influence of interreflections affect the proposed
method.
In this experiment, Object E, shown in Figure 3.12, which is separated into Regions
I and II, was used. Two setups were examined. In the first setup, the same white
paper having a uniform diffuse reflectance surface is pasted to both regions (Setup1). In the second setup, the same glossy paper having a uniform reflectance surface
is pasted up to both regions (Setup 2). In both setups, Object E was placed on thetable obliquely, as shown in Figure 3.13, so that the influence of interreflections could
be observed. If interreflections occur, it is expected that the estimated value in the
inverse local rendering method will exhibit incorrect results in this area. The resultsfor diffuse (Setup 1) and specular (Setup 2) surfaces are shown in Figures 3.14 and
3.15, respectively. Each graph represents the RGB channels of the diffuse reflectanceparameter estimated in both setups. The horizontal axis in the graphs indicates the
position of the surface point in the vertical direction of Object E, and the vertical axis
indicates the average diffuse reflectance parameter in the horizontal direction of the44

Region I

Region II

Figure 3.12. An object with interreflections (Object E).

Object E

Laser rangefinder

Figure 3.13. Position of Object E in the experiment.

object. As shown in Figure 3.14, the diffuse reflectance parameters estimated using
the present method are large around the boundary between regions 0 and 1 . Moreover,the influence of interreflections also remains in the glossy surface, as is noticeable
in Figure 3.15, because interreflections due to the specular reflection also occur atthe boundary between Regions I and II in the second setup. In this experiment, theinfluence of interreflections is actually observed to cause errors in estimation of theobject surface reflectance properties.
45

Average diffuse reflectanceparameter

150
140
130
120
110

Region I

Region II

100
1

21

41

61 81 101 121 141 161 181 201 221
Vertical posit ion of t he pixel

Average diffuse reflectanceparameter(a) R channel
150
140
130
120
110

Region I

Region II

100
1

21

41 61

81 101 121 141 161 181 201 221

Vertical position of t he pixel

Average diffuse reflectanceparameter(b) G channel
150
140
130
120
110

Region I

Region II

100
1

21

41

61

81 101 121 141 161 181 201 221

Vert ical posit ion of t he pixel(c) B channel
Figure 3.14. Estimated diffuse reflectance parameter in Experimental Setup 1.

46

Average diffuse reflectanceparameter

150
140
130
120
110

Region I

Region II

100
1

21

41

61

81 101 121 141 161 181 201 221

Vertical posit ion of t he pixel

Average diffuse reflectanceparameter(a) R channel
150
140
130
120
110

Region I

Region II

100
1

21

41 61

81 101 121 141 161 181 201 221

Vert ical posit ion of t he pixel

Average diffuse reflectanceparameter(b) G channel
150
140
130
120
110

Region I

Region II

100
1

21

41

61

81 101 121 141 161 181 201 221

Vert ical posit ion of the pixel(c) B channel
Figure 3.15. Estimated diffuse reflectance parameter in Experimental Setup 2.

47

3.5. Conclusions
In this chapter, a method was proposed for densely estimating non-uniform reflectanceproperties for almost the entire object surface using the laser rangefinder for virtualizing real objects. In the proposed approach, multiple light source positions aroundthe laser rangefinder were automatically selected, as described in Chapter 2, so thatthe diffuse and specular reflection components are observed densely. The experiments
revealed that the proposed method is useful for estimating the reflectance parametersof objects that exhibit non-uniform surface reflectance. However, some errors remain
due to both diffuse and specular interreflections. The following chapters will address
this problem by employing global illumination models.

48

Chapter 4
Inverse Radiosity for Dense
Reflectometry Estimation

4.1. Introduction

Most real object surfaces exhibit specular as well as diffuse reflection. Recoveringboth diffuse and specular reflectance models simultaneously in a mutual illumination
environment is complicated. This chapter describes the reflectance estimation method
using radiosity [ICG86] in order to consider interreflections. The radiosity rendering
method considers and renders the effect of diffuse interreflections. Hence, inverse
radiosity rendering can estimate surface reflectance parameters without the influenceof diffuse interreflections. Note that, in this chapter, interreflections are assumed to be
influenced by only the diffuse reflection component, and not by the specular reflection
component. In the proposed method, pixels that were judged in Chapter 2 to have only
the diffuse reflection component are used in the estimation of the diffuse reflectanceparameter considering diffuse interreflections. In other words, the situation can be
simplified such that the surface has only diffuse reflection components. In this case,the global illumination problem is simplified considerably and can be treated in theradiosity framework.
49

4.2. Radiosity rendering method
In most shading algorithms (i.e. the Torrance-Sparrow model), light sources have always been treated separately from the surfaces they illuminate. In contrast, radiosity
methods [ICG86] allow any surface to emit light, and so all light sources are modeled
inherently as having area. In the radiosity method, the environment is broken up intoa finite number  of discrete patches, each of which is assumed to be of finite size,
emitting and reflecting light uniformly over its entire area. If each patch is considered
to be an opaque Lambertian diffuse emitter and reflector, then, for surface ,






  










 







2 

 Ù Ú

(4.1)

3 ! !

(4.2)

 
  are the radiosities of patches  and  , respectively,
where  and  
measured in energy/unit time/unit area (i.e., /.  ). Note that  is the number ofobject patches.  is the rate at which light is emitted from patch  and has the sameunits as radiosity.  is the reflectivity of patch  and is dimensionless.
In addition,  is called the dimensionless form factor, which specifies the fractionof energy leaving the entirety of patch  that arrives at the entirety of patch , taking
into account the shape and relative orientation of both patches and the presence of any
obstructing patches (See, Figure 4.1),  and  are the areas of patches  and  ,respectively,  is the angle that the ray makes with the normal 4  of  ,  is theangle that the ray makes with the normal 4  of  , and is the length of the ray.
Moreover, 3 is either 1 or 0, depending on whether !  is visible from ! .
Figure 4.1 shows that the form factor from differential area !  to differential area! is
!





2 



3 ! *(4.3)

Determining  , the form factor from differential area !  to finite area  , requires
integration over the area of patch  as follows:


!




Ú

2 



3 ! *(4.4)

Finally, the form factor represented as Equation (4.2) from   to  is the area averageof Equation (4.4).
50

Patch v

Av

θv

r

Nu

Nv

θu

Differential area of patch u

Au
Patch uFigure 4.1. Computing the form factor between a patch and a differential area.
Equation (4.1) states that the energy leaving a unit area of surface is the sum of
the light emitted plus the light reflected. The reflected light is computed by scaling thesum of the incident light by the reflectivity. The incident light is in turn the sum of thelight leaving the entirety of each patch in the environment scaled by the fraction of thatlight reaching a unit area of the receiving patch. Here,    represents the amountof light leaving a unit area of   that reaches all of  . Therefore, multiplication by
the area ratio  . is necessary in order to determine the light leaving all of   thatreaches a unit area of  .
Conveniently, the following simple reciprocity relationship holds between form
factors in diffuse environments:   *(4.5)

Thus, Equation (4.1) can be simplified to yield  

51





  *(4.6)

This equation is referred to as the radiosity equation. In this equation, the form factor  is required. A number of studies have been developed for calculating form
factor. Cohen et al. [CG85] adapted an image-precision visible-surface algorithm toefficiently approximate form factors for occluded surfaces. Nusselt [SH81] has shownthat computing a form factor   is equivalent to projecting those parts of   that are
visible from ! onto a unit hemisphere centered about ! , projecting this projectedarea orthographically down onto the unit circle base of the hemisphere, and dividing by
the area of the circle (See, Figure 4.2). Projection onto the unit hemisphere accounts
 .  in Equation (4.4). Projection down onto the base corresponds to multifor
 . Finally, division by the area of the unit circle accounts for 2 in theplication bydenominator. Cohen et al.[CGIB86] also developed an efficient image-precision algorithm that projects patch  onto the upper half of a cube centered about !  , with the
top of the cube being parallel to the surface (Figure 4.3). Each face of this hemicube is
divided into a number of equal-sized square cells. All of the other patches are clippedto the view-volume frusta defined by the center of the cube and each of its upper fivefaces, and each of the clipped patches is then projected onto the appropriate face of thehemicube.

52

Patch v

Av

θv

Hemisphere

r

Nu

Nv

θu

Projected patch v onto hemisphere

dAu
Differential area of patch u
Figure 4.2. Hemisphere method for efficient calculation of form factor.

Patch v

Av

θv

r
Nu

Nv
Projected patch v onto hemicube

θudAu

Differential area of patch u

Figure 4.3. Hemicube method improving the hemisphere method.

53

4.3. Estimation of surface reflectance parameters considering diffuse interreflections
Since in the present study, the object is measured in a dark room, the influence ofenvironmental light can be ignored. In other words, the form factor between eachpatch of the object and the wall of the dark room need not be calculated.
Recall that Figure 2.1 shows a general flow diagram of the estimation of surface
reflectance properties by the proposed method. The proposed method consists of fourparts. The first three processes (A, B and C) have already been described in Chapter
2, and involve the measurement of an object (A and C), and the selection of a lightsource (B), respectively. In this chapter, process (D) involves inverse radiosity rendering for estimating the diffuse reflectance parameter and the estimation of the specularreflectance and surface roughness parameters using the Torrance-Sparrow model.
Multiple texture images are obtained with selected optimum light positions as described in Chapter 2. Each pixel is also classified into one of three types:  , or  .  indicates a pixel containing only the diffuse reflection component, and indicates a pixel containing a strong specular reflection component. In addition, indicates a pixel that is classified as neither 
nor  and that is not used toestimate reflectance parameters.

4.3.1 Estimation of diffuse reflectance parameter based on radiosity
Figure 4.4 shows the details of the inverse radiosity rendering method. The proposed
method has three processes. In the proposed method, the hemicube method [CGIB86]
can be used for calculating the form factor   because the object shape has alreadybeen measured by laser rangefinder. Since the range and texture images are registeredat each pixel, the radiosity  of patch  is calculated based on the sum of the valuesof the pixels that correspond to patch . Here,  represents the diffuse reflectanceparameter on patch . Then, the diffuse reflectance parameter   is determined asfollows:      

54

(4.7)

Range datam surface texture images(a) Calculation of form factor of object(b) Estimation of diffuse reflectance parameter based on
inverse radiosity rendering
Diffuse reflectance parameter
without an influence of interreflections(c) Estimation of the specular reflectance and the surface
roughness parameters using the Torrance-Sparrow modelSpecular reflectance and the surface roughness parameter

Figure 4.4. Flow diagram of estimating surface reflectance properties based on inverse radiosity rendering.
where  is 0, because there is no emission at patch  on the object. Finally, the diffuse
reflectance parameter at each surface point is estimated by calculating the averageamong neighboring patches that share the point.
Here, the calculation of the radiosity   of patch  is described in further detail.
Each patch consists of four points, and  is represented as the sum of the color valuesof the pixels that correspond to these points. Let us suppose that the patch  contains
pixels  and . It should be noted that the pixel values ) and ) of  and  in  are
obtained with different light positions,  and , respectively, because the lightsource attached to the rangefinder moves during measurement. The calculation of  requires the use of the color value ) of the pixel  with the light position . From
Lambert’s law, the color value ) can be obtained by:
)

where



Æ Ä   )
Æ Ä


  



  

(4.8)

Æ and Æ are normal vectors at pixels  and . Ä and Ä are light vectors
55

Laser rangefinderp(t)Objects u tp(s)Figure 4.5. Calculation of radiosity.

Æ Ä, thewith the light position  at pixels  and . Note that in the case ofcolor value ) should be 0 in order to consider the measurability of light reflection.

4.3.2 Estimation of specular reflectance and surface roughness parameters based on the Torrance-Sparrow model
The specular reflectance and the surface roughness parameters are estimated using theTorrance-Sparrow model as:




+
,




 




)






*

(4.9)

The specular reflectance parameter  and the surface roughness parameter  atthe surface point  are estimated by solving the simultaneous equations of Equation(4.9). In this case, the pixel values of the specular reflection components extracted fromthe two images taken under two different illumination conditions and the previously
estimated  were used.
  

   

   

(4.10)

With respect to the attenuation coefficient of a point light source + .  , the attenuationby the distance between a certain object surface point and the point light source is set
56

as unity, and the attenuation with respect to the rest of the surface points is calculated.
Note that the  estimated above should be scaled to Equation (4.9) before computing
the specular and surface roughness parameters.
  











 )
.  ). (4.11)

When strong specular reflection is observed less than twice, the specular reflectanceand surface roughness parameters cannot be determined, and thus are obtained using
a linear interpolation method within a /  / window, as described in Chapter 3. If
the window is small compared with the area that includes the undetermined reflectanceparameters, then the values of the reflectance parameters are interpolated linearly byscanning the texture image horizontally.

57

4.4. Experimental results
Objects A  E, which were used in the previous experiments, are used again for thepresent experiment. Here, Object E is used in a preliminary experiment to demonstratethat the proposed inverse radiosity method can eliminate the influence of diffuse interreflections. Note that with respect to the window size used in interpolation method,/   is fixed.
This experiment is conducted in order to confirm the effect of eliminating the influence of interreflections. Object E, having a right-angle fold, is separated into two
regions, Regions I and II, as shown in Figure 2.11(a). For this experiment, the two
setups described in Section 3.4 were used. The proposed inverse radiosity method wasthen compared to the previous method [MTY01b, MTY01a, MTY02], described in the
Chapter 3, which does not consider interreflections in estimating reflectance parameters. If interreflections occur, when the value estimated by the previous method should
be incorrect in that area. The results for diffuse (Setup 1) and specular (Setup 2) surfaces are shown in Figures 4.6 and 4.7, respectively. Each graph represents the RGB
channels of the diffuse reflectance parameter estimated by the respective methods. The
horizontal axis indicates the position of the surface point in the vertical direction of Object E, and the vertical axis indicates the average diffuse reflectance parameter in thehorizontal direction. Figure 4.6 shows that the diffuse reflectance parameters estimatedby the previous method are large around the boundary between Regions I and II. In contrast, the diffuse reflectance parameters estimated by the present method are reducedin this area, indicating the effectiveness of the inverse radiosity method. However, aslight influence of interreflections remains for the glossy surface, as can be observedin Figure 4.7, because interreflections due to the specular reflection also occur at theboundary between Regions I and II in Setup 2. The diffuse reflectance parameter obtained by the present method looks smoother than that obtained by the previous method
because the diffuse reflectance parameter of a point on the object is calculated as theaverage of the diffuse reflectance parameters of four patches that share the point.

58

Average diffuse reflectanceparameter

150
Previous method

140
130
120
110

Present method

Region IA

RegionRegionBII

100
1

21

41

61

81 101 121 141 161 181 201 221

Vertical posit ion of t he pixel

Average diffuse reflectanceparameter(a) R channel
150
Previous method

140
130
120
110

Present method

RegionAI
RegionRegionBII
Region

100
1

21

41

61

81 101 121 141 161 181 201 221

Vertical position of t he pixel

Average diffuse reflectanceparameter(b) G channel
150
Previous method

140
130
120
110

Present method

Region I
Region A

Region II
Region B

100
1

21

41

61

81 101 121 141 161 181 201 221

Vert ical posit ion of the pixel(c) B channel
Figure 4.6. Comparison with previous results for Object E with diffuse surfaces.

59Average diffuse reflectanceparameter

150
Previous method

140
130
120
110

Present method

Region I

Region II

100
1

21

41

61

81 101 121 141 161 181 201 221

Vertical posit ion of t he pixel

Average diffuse reflectanceparameter(a) R channel
150

Previous method

140
130
120
110

Present method

100
1

21

41 61

Region I

Region II

81 101 121 141 161 181 201 221

Vert ical posit ion of t he pixel

Average diffuse reflectanceparameter(b) G channel
150
Previous method

140
130
120
110

Present method

Region I

Region II

100
1

21

41

61

81 101 121 141 161 181 201 221

Vert ical posit ion of the pixel(c) B channel
Figure 4.7. Comparison with previous results for Object E with glossy surfaces.

60(a) Object A(b) Object B(c) Object C(d) Object D

Figure 4.8. Difference between diffuse reflectance parameters estimated with inverse
radiosity and those estimated with the method in Chapter 3.

4.4.1 Estimated reflectance parameters
The following experiment was conducted in order to examine the effect of considering interreflections in reflectance parameter estimation of non-uniform objects, i.e.,
Objects A, B, C and D. Figures 4.8(a)  (d) illustrate the sum of the differences in
RGB channels of the diffuse reflectance parameters between the previous and present
methods. These images are shown in gray-scale using cylindrical coordinates having amaximum value of 255. The difference is large in areas in which interreflections occur,
indicating that the present method is effective for objects that have non-uniform surface
reflectance properties and exhibit interreflections. Some areas have larger differences
than others. For example, the area of the beak of Object A and the arms and legs of
Objects B and C have remarkable differences. In the inverse radiosity method, the diffuse reflectance parameter is calculated as the average of four patches, as explained forthe preliminary experiments. Therefore, the diffuse reflectance parameter that includes
the influence of specular interreflections is also smoothed, and this smoothing processcauses a large difference, compared to the previous method.
61

4.4.2 Rendering results using the radiosity method and the TorranceSparrow model
Finally, the virtual relighting of virtualized objects is demonstrated. Figures 4.9 4.12
show the rendering results of virtualized objects (Objects A, B, C and D) under a virtual
lighting condition. A virtual light source is rotated around the virtualized object, andthe center of rotation is at the center of the object. Each figure shows the movement of
the light source from right to left of the object. Each object is fairly rendered under arbitrary virtual illumination conditions without the influence of diffuse interreflections.
However, because specular interreflections cannot be removed, complicated areas of
Objects C and D contain some highlights compared with other areas. Areas that are
surrounded by color squares in Figure 4.13 show different result between the inverse
local rendering and the inverse rendering. For example, the diffuse reflection component differs between the left and right images in (a). The rendering result of inverse
radiosity rendering shows a reduction in the influence of diffuse interreflections compared to the result of inverse local rendering, and (b) shows that the influence of diffuse
interreflections is eliminated.

62

Figure 4.9. Rendering results for Object A with the Torrance-Sparrow model.

Figure 4.10. Rendering results for Object B with the Torrance-Sparrow model.

Figure 4.11. Rendering results for Object C with the Torrance-Sparrow model.

Figure 4.12. Rendering results for Object D with the Torrance-Sparrow model.

63

Inverse local rendering Inverse radiosity rendering(a) Object A

Inverse local rendering

Inverse radiosity rendering(b) Object B
Figure 4.13. Difference between rendering results based on the inverse radiosity rendering and the inverse local rendering in Chapter 3.

64

4.5. Discussion
Again, the preliminary experiments are discussed, particularly Experimental Setup 2.
In Setup 1, it is proved that the inverse radiosity method can remove the influence ofdiffuse interreflections because the radiosity rendering method can take the diffuse interreflections into account. However, it should be noted that the radiosity rendering
method considers only diffuse interreflections. Due to this limitation, in Experimental
Setup 2, some errors remain. There are two possible approaches by which to address this problem. The first is based on hybrid rendering, such as the combinationof radiosity and ray tracing rendering methods [WCG87, Shi90, Arv86]. This method
has a greater computational cost compared to single rendering methods. Therefore,
the inverse hybrid rendering method is computationally very expensive for estimating
surface reflectance properties and does not provide a solution to the above-mentioned
problem. The second method is based on a more efficient rendering method, which canrepresent both diffuse and specular interreflections, that has recently been developed.
The usefulness of such a rendering method has been proven in the field of computer
graphics. This method, called the photon mapping rendering method [Jen01], is computationally less expensive than the hybrid rendering method.
Moreover, in the inverse radiosity method, the object must be separated into patches
for calculation of the form factor. In order to address this problem, the hierarchicalconstruction of patches has been proposed [WEH89]. This method reduces the computational cost by constraining the subdivision of patches according to surface normal.
However, a limitation exists in the finite element separation, and the method becomesvery costly for complex models and non-diffuse materials. The high cost of this method
for complex models is due to the fact that the radiosity algorithm computes values forevery patch in the model. Furthermore, the radiosity algorithms represent the solution
in a finite mesh, a tessellated representation of the real geometry. The representation
is inaccurate if the mesh is not carefully constructed, and, as a result, radiosity algorithms have problems computing sharp features in the illumination. For example,
sharp shadow boundaries tend to be blurred. In contrast to radiosity, photon mapping
rendering does not require the separation of an object into patches. This method is
performed using a pixel in the scene image from the camera. Due the nature of thismethod, lighting effects are physically represented. If inverse photon mapping can be
performed, more accurate surface reflectance parameters can be obtained.
65

4.6. Conclusions
In this chapter, a new surface reflectance modeling method was proposed that is basedon densely estimating the non-uniform reflectance properties with diffuse interreflections for almost the entire surface of objects using a laser rangefinder in order to virtualize real objects. In this approach, the radiosity rendering method was employedso that diffuse interreflections can be considered when estimating surface reflectanceproperties. The present method was shown experimentally to be useful for estimating the reflectance parameters of objects with diffuse interreflections that exhibit nonuniform surface reflectance. However, a problem remains with respect to the influenceof specular interreflections. Because the radiosity method considers only diffuse interreflections, the reflectance parameters may be influenced by specular interreflections.
In order to investigate this possibility, a preliminary experiment was conducted using asimple object. It was confirmed experimentally that if specular interreflections occur,then the estimated parameter contains some errors. In the next chapter, a method forhandling the problem of both diffuse and specular interreflections will be introduced.
Specifically, a global illumination rendering method called photon mapping will be
employed, in which all lighting effects in the real world are taken into account. Using
this method, faithful diffuse reflectance, specular reflectance and surface roughness
parameters will be obtained.

66

Chapter 5
Inverse Photon Mapping for Dense
Reflectometry Estimation
5.1. Introduction
As described in the previous chapter, the radiosity algorithm is quite efficient at computing the lighting distribution with respect to a simple model with diffuse materials.
However, because the radiosity rendering method considers only diffuse interreflections, when object reflectance properties are estimated using the inverse radiosity rendering, it is impossible to eliminate the influence of specular interreflections. In addition, computation becomes very costly for complex models and non-diffuse materials.
The high computational cost for complex models is due to the fact that the radiosity
algorithm computes values for every patch in the model. Furthermore, the radiosity
algorithms represent the solution in a finite mesh, and a tessellated representation canbe very inaccurate if the mesh is not carefully constructed. Therefore, accurate object
geometry and efficient tessellation are required when using the radiosity to estimate
surface reflectance properties.
In this chapter, the photon mapping rendering method is used to solve the problem with the inverse radiosity rendering method. Using the inverse photon mapping
method, both diffuse and specular interreflections on the object surface can be taken
into account. Because the photon mapping rendering method does not require tessellated patches, due to its pixel base calculations, the computational cost is low comparedwith the radiosity rendering method. The photon mapping method is introduced in the67

following.

5.2. Photon mapping
For rendering both diffuse and specular interreflections, hybrid techniques exist thatcombine the radiosity and ray-tracing methods. Radiosity is good for treating diffuse
reflections, whereas ray tracing is good for treating specular reflections. Hybrid methods use ray tracing to add specular reflections to radiosity [WCG87, Shi90, Arv86,
CRMD91, RPV93, ZS95]. However, representing specular reflection requires more
sampling rays after radiosity rendering has been applied to the object, which results ina large computational cost.
Photon mapping, on the other hand, involves a different approach from the hybrid
techniques. The idea in photon mapping is to change the representation of illumination. Rather than tightly coupling lighting information with the geometry, the lighting
information is stored in a separate independent data structure called the photon map[Jen01]. The photon map is constructed of photons emitted from the light sources andtraced through the reflectance model (i.e. the Torrance-Sparrow model). The photon
map contains information about all of the photon hits, and this information can beused to efficiently render the model in a manner similar to that in which the radiosity
is used in hybrid techniques. The decoupling of the photon map from the geometryoffers a significant advantage in that the representation is not only simplified, but it isalso possible to use the structure to represent lighting in very complex models. The
combination of photon mapping and a Monte Carlo ray-tracing based rendering algorithm results in an algorithm that is as general as pure Monte Carlo ray tracing butthat is significantly more efficient [Kaj86]. In fact, the calculation cost of the Monte
Carlo ray-tracing method grows exponentially with respect to the number of rays as
the number of reflections increases. For example, if ten rays are used to compute irradiance, and each of these ten rays intersect another diffuse surface, which requires
another ten rays to compute the irradiance, and as a result, 100 rays are required inorder to compute the diffuse light reflected twice before being seen by the observer.
In the photon mapping rendering method [Jen01], an outgoing radiance " from asurface point  is calculated in order to decide the surface color. Figure 5.1 shows
the geometrical arrangement for photon mapping rendering. The following equations
68

L

r
n
dω

Projected area of dA

x
Differential area of point xdA

Figure 5.1. Geometrical arrangement for photon mapping rendering.

form the rendering equations in the photon mapping method.

  "     "  

" 

 
"  #  

¼


#  

¼(5.1)

"  

 
 


 !
¼

¼

¼

(5.2)

  #     #  
 

¼

¼

(5.3)


¼  "  
¼  "  
 ¼   "  
¼
"  

(5.4)

, "  
, "  
¼  and #  ¼ 
 represent the emitted radiance,
where "  
the reflected radiance, the incoming radiance, and a BRDF (i.e. the Torrance-Sparrow¼ 
 and #  ¼ 
 denote the diffuse and specularmodel), respectively. #  
¼  represents the direct illumination from theterms in BRDF, respectively. "  
¼ is referred to as the caustics, which represent the indirect
light sources. "  
 ¼
illumination from the light sources via specular reflection or transmission. "   
denotes the indirect illumination from light sources that have been reflected diffusely atleast once. Equations (5.1)  (5.4) indicate that the reflected radiance has the following
four components.
 "  #  

"  

 
 


 !

¼



¼


#  

¼

"  

 
 

 
 !

¼



¼


#  

¼

¼

¼

¼

(5.5)

"     "   
 

 
 !
¼

69

¼

¼

¼

(5.6)

(a) Monte Carlo ray tracing
method(b) Photon mapping method(10,000 photons)(c) Photon mapping method (d) Photon mapping method(100,000 photons)(1,000,000 photons)
Figure 5.2. Rendering results of photon mapping.
#  "  

 
 

 
 !

(5.7)


#  "  

 
 


 !

(5.8)

¼



¼

¼

¼

¼

¼

¼

¼

where Equations (5.5)(5.8) represent the direct illumination, specular and glossy reflection, caustics and multiple diffuse reflections terms, respectively. This equation isused whenever the reflected radiance from a surface is computed. Figure 5.2 shows therendering results obtained using the Monte Carlo ray tracing method and the photon
mapping method. In (a), the rendering result has some noise due to the small numberof rays cast. On the other hand, in (b)  (d), the greater the number of distributed photons, the lower the sampling noise level. Based on these results, the photon mapping70

method can efficiently render the virtual image.

5.3. Estimation of surface reflectance parameters considering both diffuse and specular interreflections
Here, the outgoing radiance " in Equation (5.1) is equivalent to the reflected radiance" due to the assumption that the underlying objects have no emissions. Equations(5.1) and (5.2) are theoretical models. Using Equation (2.9), the color  at surface
point  is represented by the following equation, which is referred to as the Ward
reflectance model [War92]:




 



 


2

2

 

)& . 2  5    


(5.9)

where  is the incoming radiance. 5     denotes the specular term in Equation (2.12), and the other parameters are the same as in Equation (2.12). In practice,
the Ward model described above has five parameters for representing anisotropic object surface reflectance properties. Because the object is assumed to have isotropic
reflectance properties, there are three unknown parameters: the diffuse reflectance, thespecular reflectance and the surface roughness parameters.  is decided by counting
the number of photons that arrive at the point . The photon is specifically traced using a Monte Carlo ray tracing method [Kaj86]. In this case, the photon is reflected orabsorbed according to the reflectance parameters, and only the photon that is reflected
is traced iteratively.
Figure 5.3 shows the flow diagram of the present method. This figure illustrates
process (D) in Figure 2.1 as described in Chapter 2. As an initial estimation, the reflectance parameters are obtained using a previous method based on inverse radiosity
rendering [MY02b, MY02a] in Figure 5.3(a). In this preprocessing, the diffuse reflectance parameter is estimated based on the inverse radiosity in order to consider
diffuse interreflections. The specular reflectance and the surface roughness parameters
are estimated based on the inverse local rendering (i.e. the Torrance-Sparrow model)
with no consideration of specular interreflections. Here, let    ,   and   be the71Range datam surface texture images(a) Initial estimation of the diffuse reflectance, specular reflectanceand surface roughness parameters by the inverse radiosity method
Initial diffuse reflectance, specular reflectanceand surface roughness parameters(b) Estimation of specular reflectance and surface roughness
parameters using the photon mapping rendering
Diffuse reflectance, specular reflectanceand surface roughness parameter

Figure 5.3. Flow diagram for the estimation of surface reflectance properties based on
inverse photon mapping.

reflectance parameters obtained in this process. These parameters are used as initial
parameters for the next process (b) in Figure 5.3.
Figure 5.4 illustrates the details of the iterative reflectance estimation process basedon inverse photon mapping in Figure 5.3 (b). In the following, the details of reflectanceparameter estimation are described.
The reflectance parameter estimation method based on inverse photon mapping isseparated into two processes, (a) and (b), shown in Figure 5.4. The first process is forthe diffuse reflectance parameter estimation ((a) in Figure 5.4), and the second process
is for the estimation of the specular reflectance and surface roughness parameters ((b)
in Figure 5.4). These processes are performed iteratively. In each process, the following equation, which is derived from Equation (5.9), is minimized at each pixel in the
texture image:    









   

72

Initial reflectance parameters (Pdx0 ,Psx0

Range data, texture images,σ x0 )(a) Estimation of diffuse reflectance parameter
Rendering based on Photonmap
Estimation of diffuse reflectance parameter Pdx
using BRDF modelnUpdate of reflectance parameter Pdx
nReflectance parameters ( Pdx , Psxn −1, σ xn −1 )n-th iteration(b) Estimation of specular reflectance and surface roughness parameter
Rendering based on Photonmap
Estimation of specular reflectance parameter Psx andsurface roughness parameter σ x using BRDF modelnUpdate of reflectance parameters Psx and σ xnn

Reflectance parameters ( Pdx , Psxn , σ xn )Figure 5.4. Detail of reflectance estimation process based on inverse photon mapping.







" 


2



   5    (5.10)

where  is the measured radiance (color intensity) at surface point  with light source
position ,  is the irradiance that is computed from Equation (5.9) at surface point
 with light source position , and  denotes the number of selected light positions at
surface point  in categories  ,  and  among the selected light positions.
In process (a), the diffuse reflectance parameter   is estimated using a pixel thatis categorized as  .   ,   and   are used to compute  only at the first73

iteration. Here, the specular reflection term in Equation (5.9),   5   , isset to be 0 because the specular reflection cannot be observed.
In process (b), the specular reflectance  and the surface roughness  parameters
are estimated using only pixels that are categorized as  or  .   and   areagain used to compute  only at the first iteration. The  estimated above is used in
Equation (5.9). When  and  are estimated, the value of each reflectance parameteris updated, and processes (a) and (b) are iterated  times. The reflectance parameteris selected when differences between the real and synthetic images is the minimum
value among  samples. Because this is a non-linear equation with unknown parameters, the photon mapping rendering and estimation of surface reflectance parameters
is performed iteratively, and the difference between the real image and the synthesizedimage is minimized (Equation (5.10)). A number of methods can be used to minimize
this equation, and the downhill simplex method is selected for this problem [BG01].
After the estimation process is complete, the specular reflectance and the surface
roughness parameters may not be correct, if the specular reflection component is exceedingly small. Such parameters are interpolated linearly by scanning the texture
image horizontally.

74

5.4. Experimental results
In Chapter 3 and 4, two methods of inverse reflectometry (hereinafter referred to
as Method I and Method II) were proposed. In the following, using Object E, thereflectance parameters estimated using the present method (hereafter referred to as
Method III) are compared with those obtained by Methods I and II. The effect of considering interreflections in surface reflectance parameter estimation is then examinedusing Objects A  D with uniform and non-uniform surface properties. Finally, rendered images are obtained based on reflectance parameters estimated using Method III.
See Figure 2.11 and 3.12 for the test objects used in these experiments.
A standard PC (Pentium 4, 3.06 GHz, memory: 2 GB) is used in the following
experiments. The number of photons is 2 million, and the proposed algorithm requires
approximately four hours to estimate the reflectance parameters of each object. The
threshold is fixed at   .

5.4.1 Preliminary experiments
The performance of the present method was demonstrated in preliminary experiments
using a simple object (Object E). In particular, Method III was compared with Methods
I (which does not consider interreflections, as described as in Chapter 3) and II (which
considers only diffuse interreflections, as described as in Chapter 4). Object E consistsof two plates (Regions I and II) situated at a 90-degree angle with respect to eachother. The same two experimental setups described in Section 3.4 were used, and theresults are shown in Figure 5.5 for Setup 1 and in Figure 5.6 for Setup 2. Each graphrepresents the RGB channels of the diffuse reflectance parameter estimated by the threemethods. The horizontal axis represents the position of the pixel along the verticaldirection of the object, and the vertical axis represents the average diffuse reflectanceparameter along the horizontal direction of the object. In Methods I and II, the valueof the diffuse reflectance parameter is large around the boundary between Regions I
and II due to the influence of interreflections. However, in Method III, the estimatedparameter is more stable, indicating that Method III can eliminate the influence of both
diffuse and specular interreflections.

75Average diffuse reflectanceparameter

150

Method I

140
Method II

130
120
110

Method III

Region I

Region II

100
1

21

41

61 81 101 121 141 161 181 201 221
Vertical posit ion of t he pixel

Average diffuse reflectanceparameter(a) R channel
150

Method I

140

Method II

130
120
110

Method III

Region I

Region II

100
1

21

41 61

81 101 121 141 161 181 201 221

Vertical position of t he pixel

Average diffuse reflectanceparameter(b) G channel
150

Method I

140

Method II

130
120
110

Method III

Region I

Region II

100
1

21

41

61

81 101 121 141 161 181 201 221

Vert ical posit ion of t he pixel(c) B channel
Figure 5.5. Comparison of the three proposed methods for Object E with diffuse surfaces.

76Average diffuse reflectanceparameter

150

Method I

140

Method II

130
120
110

Method III

Region I

100
1

21

41

61

Region II

81 101 121 141 161 181 201 221

Vertical position of the pixel

Average diffuse reflectanceparameter(a) R channel
150

Method I

140

Method II

130
120
110

Method III
Region I

100
1

21

41

61

Region II

81 101 121 141 161 181 201 221

Vertical posit ion of t he pixel

Average diffuse reflectanceparameter(b) G channel
150

Method I

140
Method II

130
120
110

Method III

100
1

21

41

61

Region I

Region II

81 101 121 141 161 181 201 221

Vertical position of the pixel(c) B channel
Figure 5.6. Comparison of the three proposed methods for Object E with glossy surfaces.

775.4.2 Estimated reflectance parameters
In the next experiment, Objects A  D, shown in Figures 2.11, were used. These
objects have non-uniform or uniform diffuse and specular reflectance properties. The
computational times for these methods are listed in Table 5.1. The inverse photon mapping rendering takes longer to estimate the surface reflectance properties than Methods
I or II. This is because rendering the object image based on estimated reflectance parameters requires a great deal of time.

Table 5.1. Comparison of computational costs of the three proposed methods.
Object
MethodComputational Time [h:m]
A

B

C

D

Method I
Method II
Method III
Method I
Method II
Method III
Method I
Method II
Method III
Method I
Method II
Method III

0:20
3:32
4:39
0:26
3:51
4:58
0:23
3:11
4:43
0:18
2:34
4:21

Figure 5.7  5.10 show the cylindrical images of real objects and the difference images between real and synthetic images (rendered by photon mapping) for Methods II
and III for each test object (Objects A  D), respectively. The light position is locatedat the top of the rangefinder. Synthetic images are rendered using estimated reflectanceparameters under the illumination conditions used for the real images. Note that linear interpolation is conducted when the specular reflectance and the surface roughness
parameters cannot be estimated due to an exceedingly small specular reflection value.
In Method II, the error due to the influence of specular interreflections is confirmed.
In particular, Objects A and B exhibit large errors at the areas showing inequalities
78

Table 5.2. Average and variance of differences between real and synthetic images.
Object A Object B Object C Object D
Average Method II
17.7
20.1
8.7
13.3
Method III
1.11
1.06
0.51
0.92
Variance Method II
501.9
485.4
493.3
375.2
Method III
6.8
8.0
3.2
9.8

(i.e., the cat’s leg and neck, and the pig’s nose). Method III is not affected by suchinfluences. In addition, Table 5.2 shows the variance of differences between real andsynthetic images. Method III has much smaller variances than Method II for all of theobjects. These results show that Method III can accurately estimate each reflectanceparameter, even if diffuse and specular interreflections occur.
Figure 5.11 shows the relationship between the iterated process and the differences
between real and synthetic images. The vertical axis indicates the sum of differences
between the real and synthetic images, and the horizontal axis indicates the number of
iterations. Each graph shows that the iterated estimation process decreases the difference between real and synthetic images. However, the minimum difference may not
be the global minimum because the proposed iteration method ends when the numberof iterations reaches 50. Therefore, each graph illustrates pulsation of the differences.
In other words, the higher the number of iterations, the lower the differences. In thisexperiment, the reflectance parameters of each object is determined when the numberof iterations are 34, 32, 39 and 35, respectively.

79

(a) Real image(b) Difference between real and synthetic images for Method II(c) Difference between real and synthetic images for Method III
Figure 5.7. Differences between real and synthetic cylindrical images for Object A.

80

(a) Real image(b) Difference between real and synthetic images for Method II(c) Difference between real and synthetic images for Method III
Figure 5.8. Differences between real and synthetic cylindrical images for Object B.

81

(a) Real image(b) Difference between real and synthetic images for Method II(c) Difference between real and synthetic images for Method III
Figure 5.9. Differences between real and synthetic cylindrical images for Object C.

82

(a) Real image(b) Difference between real and synthetic images for Method II(c) Difference between real and synthetic images for Method III
Figure 5.10. Differences between real and synthetic cylindrical images for Object D.

83

The difference between realand synthetic images

9000
8000
7000
6000
5000
4000
3000
2000
1000
0
1

3

5

7

9

11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

The number of iteration

The difference between realand synthetic images(a) Object A
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
1

3

5

7

9

11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

The number of iteration

The difference between realand synthetic images(b) Object B
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
1

3

5

7

9

11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

The number of Iteration

The difference between realand synthetic images(c) Object C
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
1

3

5

7

9

11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

The number of iteration(c) Object D
Figure 5.11. Relationship between the number of iterations and difference between
real and synthetic images for Objects A  D.
84

5.4.3 Rendering results obtained using the photon mapping method
Figure 5.12 shows rendered images of Objects A  D based on reflectance parameters estimated using the inverse photon mapping rendering. Although these images
are photorealistically rendered, some errors remain with respect to the geometry. For
example, some areas of the duck’s legs (a) are not rendered, and spike noise can beobserved around the cat’s legs (c). These errors are due to noise in the range images.
In order to solve this problem, it is necessary to interpolate the range data using thedata around these areas.

85

(a) Rendering results for Object A(b) Rendering results for Object B(c) Rendering results for Object C(c) Rendering results for Object D
Figure 5.12. Rendering results for Object A  D under arbitrary illumination conditions.

865.5. Discussion
Inverse photon mapping rendering has efficiently estimated the object surface reflectanceproperties. However, the method has a problem with respect to computational time.
This problem is due to the time consumed by the photon mapping rendering. Althoughphoton mapping is faster than other global illumination rendering methods, the amountof time required is still too great for rendering images by a standard PC. There are two
approaches by which to solve this problem.
The first involves simply implementing and calculating the photon mapping rendering algorithm on fast graphics hardware (GPU). Recently, the photon mapping
rendering by such methods has been reported to be much faster than calculation by
CPU[PDC 03, PBMH02]. Using these methods, the inverse photon mapping rendering will be efficiently calculated.
In the second approach, the photon map is rendered only at the areas in which thesurface reflectance properties vary. In inverse photon mapping rendering, the entire
object surface is rendered in each iteration process. However, the areas at which thereflectance parameters do not change need not be re-rendered. In order to address
this problem, in areas where the photon map does not change, the radiance of thepreviously estimated area is used as a radiance cache. Although this algorithm causes
some errors in the reflectance parameter, the degree of trade-off between accuracy andand real-time calculation can be controlled.

5.6. Conclusions
In this chapter, a new method has been proposed for densely estimating the nonuniform reflectance properties of real objects based on the inverse global rendering
technique. The photon mapping rendering method has been proposed in order to solve
the inverse rendering equations. Using the proposed method, the reflectance parameters of objects having either uniform or non-uniform surface reflectance properties
and both diffuse and specular interreflections can be estimated. In order to verify
the present method, a preliminary experiment was conducted and the present method
was shown to be able to remove the influence of both diffuse and specular interreflections. In addition, an experiment was conducted using real objects having uniform87

or non-uniform reflectance properties and interreflections. The inverse photon mapping rendering was shown to be able to estimate object reflectance parameters withno influence due to either diffuse or specular interreflections. In future research, theproposed method will be improved in order to efficiently calculate the creation of thephoton map, and the computational cost will be reduced by using a GPU. The present
method can be applied to mixed-reality applications if it works in real time. In addition, the present method will be applied to an anisotropic surface. Although the objects
examined herein had isotropic surface reflectance, applying the present method to ananisotropic surface reflectance should be possible by simply modifying the local illumination model used in this chapter.

88

Chapter 6
Conclusions
The objective of the present study was to develop a new surface reflectance modelingmethod that can densely estimate non-uniform reflectance properties for almost theentire object surface using a laser rangefinder for virtualizing real objects.
Multiple light source positions around the laser rangefinder were automatically selected, so that both diffuse and specular reflection components were observed densely.
The proposed approach was shown experimentally to discriminately observe the diffuse and specular reflection components on the object surface. However, even whenusing the proposed approach, observing both reflection components over the entire surface of an object is difficult. Therefore, the lighting conditions in the scene must beplanned in order to observe the reflection components densely.
Three methods were proposed for surface reflectance estimation based on the inverse rendering framework without the assumption of uniform object surface reflectanceproperties:

 the inverse local rendering method, the inverse radiosity method, and the inverse photon mapping method.
The first approach is based on the inverse local rendering method, in which theTorrance-Sparrow model is employed as a local illumination model, and the data ofrange and multiple images obtained by the proposed light selection method are fitted
on the Torrance-Sparrow model. The surface reflectance parameters are then estimated.
89In order to verify the applicability of the proposed inverse local rendering method, an
experiment was conducted in order to show that the reflectance parameters can be
estimated at each object point for objects with non-uniform reflectance properties. In
addition, the influence of interreflections revealed the need for estimation consideringinterreflections.
The second and third approaches are based on the inverse global rendering method
for solving the problem involved in the inverse local rendering (i.e. interreflections).
In the second approach, the radiosity rendering method, which can represent diffuse
interreflections, was employed. This method can also be used to estimate non-uniform
surface reflectance properties using multiple images obtained based on the light selection method. The advantage of the proposed method is the elimination of diffuse interreflections. However, the problem concerning the influence of specular interreflections
remains.
In the third approach, the photon mapping method, which can represent all of thelighting effects in the real world, is employed. This rendering method is rapid and has
low memory cost compared to other global illumination rendering methods. Unlike theradiosity rendering method, this method is independent of the complexity of the object
geometry. Therefore, the inverse photon mapping method can be applied to objects
with various reflectance properties. Experiments have shown that the third method
exhibits the best performance among the three methods proposed in this study. The
advantage of the inverse photon mapping method is that not only diffuse interreflections but also specular interreflections are considered in reflectometry estimation. As
a result, the estimated parameters can be obtained without the influence of both interreflections.
In future studies, the following items will be investigated further: Light planning in observing the object surface reflection component.
 Automatic estimation of scene lighting conditions.
 Real-time estimation using graphics hardware.
The first item can improve the light selection method so as to observe both object surface reflection components more accurately and densely. The second item helps tomerge the virtual and virtualized objects into the real world. Because the proposed90

method can be used to estimate object reflectance parameters, faithfully representing
the object is easy if the lighting condition is known. The third item is concernedwith rapid estimation by implementing the algorithm on high-speed graphics hardware (GPU). Photon mapping rendering has already been implemented on GPUs inother studies. Therefore, using these methods, the proposed estimation processes canbe performed in real time. Real-time estimation is extremely useful in the fields ofcomputer vision and graphics. If the three above-described items are achieved, theproposed technique can be practically applied to a number of fields, such as mixed
reality.

91

References[Arv86]

J. Arvo, “Backward Ray Tracing,” Proc. ACM SIGGRAPH ’86 Seminar
Notes, 1986.

[Azu97]

R. T. Azuma, “A Survey of Augmented Reality,” Presence, Vol. 6, No. 4,
pp. 355-385, 1997.

[BG01]

S. Boivin and A. Gagalowicz, “Image-Based Rendering of Diffuse, Specular and Glossy Surfaces from a Single Image,” Proc. ACM SIGGRAPH’01, pp. 107-116, August 2001.

[BH85]

M. J. Brooks and B. K. P. Horm, “Shape and Source from a Single Image,”
Proc. Int. Joint Conf. on Artifcial Intelligence, pp. 932-936, 1985.

[BRG92]

R. Baribeau, M. Rioux and G. Godin, “Color Reflectance Modeling Using a Polychromatic Laser Sensor,” IEEE Trans. on Pattern Analysis andMachine Intelligence, Vol. 14, No. 2, pp. 263-269, 1992.

[CG85]

M. F. Cohen and D. P. Greenberg, “The Hemi-Cube: A Radiosity Solution for Complex Environments,” Proc. ACM SIGGRAPH ’85, pp. 31-40,
1985.

[CGIB86]

M. F. Cohen, D. P. Greenberg, D. S. Immel and P. J. Brock, “An Efficient Radiosity Approach for Realistic Image Synthesis,” IEEE Trans. on
Computer Graphics and Application, Vol. 6, No. 3, pp. 26-35, 1986.

[Che95]

E. Chen, “QuickTime VR - An Image-based Approach to Virtual Environment Navigation,” Proc. ACM SIGGRAPH ’95, 1995.
93

[Coo86]

R. L. Cook, “Stochastic Sampling in Computer Graphics,” ACM Trans.
on Graphics, Vol. 5, No. 1, pp. 51-72, 1986.

[CPC84]R. L. Cook, T. Porter and L. Carpenter, “Distributed Ray Tracing,” Proc.
ACM SIGGRAPH ’84, pp. 137-145, 1984.

[CRMD91] S. E. Chen, H. E. Rushmeier, G. Miller and T. D. Derose, “A Progressive
Multipass Method for Global Illumination,” Proc. ACM SIGGRAPH ’91,
pp. 165-174, 1991.
[CSSD96]

P. H. Christensen, E. J. Stollnitz, D. H. Salesin and T. D. DeRose, “Global
Illumination of Glossy Environments Using Wavelets and Importance,”
ACM Trans. on Graphics, Vol. 15, No. 1, pp. 37-71, 1996.

[DM96]

P. E. Debevec and C. J. Malik, “Modeling and Rendering Architecturefrom Photographs: A Hybrid Geometry- and Image-based Approach,”Proc. ACM SIGGRAPH ’96, pp. 11-20, 1996.

[DM03]

D.Samaras and D. Metaxas, “Incorporating Illumination Constraints in
Deformable Models for Shape from Shading and Light Direction Estimation,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 25,
No. 2, pp. 247-264, 2003.

[DYB98]

P. Debevec, Y. Yu and G. Borshukov, “Efficient View-dependent Imagebased Rendering with Projective Texture-mapping,” Proc. Eurographics
Workshop on Rendering, pp. 105-116, 1998.

[FGR93]

A. Fournier, A. S. Gunawan and C. Romanzin, “Interactive Common Illumination for Computer Augmented Reality,” Proc. Int. Conf. on Graphics
Interface (GI1993), pp. 254-262, May 1993.

[FvDFH93] James Foley, Andries van Dam, Steven Feiner and John Hughes, Computer Graphics Principles and Practice, Addison-Wesley Publishing
Company, 2nd edition, 1993.
[HA93]

D. Hougen and N. Ahuja, “Estimation of the Light Source Distribution
and Its Use in Integrated Shape Recovery from Stereo and Shading,” Proc.
IEEE Int. Conf. Computer Vision(ICCV1993), pp. 148-155, 1993.
94

[Hor90]

B. K. P. Horm, “Height and Gradient from Shading,” Int. Journal of Computer Vision, Vol. 5, No. 1, pp. 584-595, 1990.

[ICG86]

D. S. Immel, M. F. Cohen and D. P. Greenberg, “A Radiosity Method for
Non-diffuse Environments,” Proc. ACM SIGGRAPH ’86, pp. 133-142,
1986.

[IS91]

K. Ikeuchi and K. Sato, “Determining Reflectance Properties of an Object
Using Range and Brightness Images,” IEEE Trans. on Pattern Analysis
and Machine Intelligence, Vol. 13, No. 11, pp. 1139-1153, 1991.

[Jen95]

H. W. Jensen, “Importance Driven Path Tracing Using the Photon Map,”
Proc. Eurographics Rendering Workshop, pp. 326-335, 1995.

[Jen01]

H. W. Jensen, Realistic Image Synthesis Using Photon Mapping, A K
Peters, Ltd, 1st edition, 2001.

[Kaj86]

J. T. Kajiya, “The Rendering Equation,” Proc. ACM SIGGRAPH ’86,
pp. 143-150, August 1986.

[KC94]

G. Kay and T. Caelli, “Inverting an Illumination Model from Range andIntensity Maps,” CVGIP: Image Understanding, Vol. 59, No. 2, pp. 183201, 1994.

[LDR00]C. Loscos, G. Drettakis and L. Robert, “Interactive Virtual Relighting ofReal Scenes,” IEEE Trans. Visualization and Computer Graphics, Vol. 6,
No. 4, pp. 289-305, December 2000.

[LF94]

S. Laveau and O. Faugeras, “3-D Scene Representation as a Collection of
Images,” Proc. Int. Conf. on Pattern Recognition (ICPR1994), pp. 689691, 1994.

[LH96]

M. Levoy and P. Hanrahan, “Light Field Rendering,” Proc. ACM SIGGRAPH ’96, pp. 31-42, 1996.

[LL95]

J. Lu and J. Little, “Reflectance Function Estimation and Shape Recovery
from Image Sequence of a Rotating Object,” Proc. 5th IEEE Int. Conf. on
Computer Vision, pp. 80-86, June 1995.
95

[LL99a]S. Lin and S. W. Lee, “A Representation of Specular Appearance,” Proc.
7th IEEE Int. Conf. on Computer Vision, Vol. 2, pp. 849-854, 1999.

[LL99b]S. Lin and S. W. Lee, “Estimation of Diffuse and Specular Appearance,”
Proc. 7th IEEE Int. Conf. on Computer Vision, Vol. 2, pp. 855-860, 1999.

[LLSS03]

H. P. A. Lensch, J. Lang, A. M. Sa and H. P. Seidel, “Planned Sampling of
Spatially Varying BRDFs,” Proc. Eurographics ’03, pp. 473-482, 2003.

[LR85]

C. H. Lee and A. Rosenfeld, “Improved Methods of Estimating Shapefrom Shading Using the Light Source Coordinate System,” Proc. Int.
Joint Conf. on Artificial Intelligence, Vol. 26, pp. 125-143, 1985.

[LW95]

E. P. Lafoune and Y. D. Willems, “A 5D Tree to Reduce the Varianceof Monte Carlo Ray Tracing,” Proc. Eurographics Rendering Workshop,
pp. 11-20, 1995.

[MB95]

L. McMillan and G. Bishop, “Plenoptic Modeling: An Image-based Rendering System,” Proc. ACM SIGGRAPH ’95, 1995.

[MMMS01] Y. Mukaigawa, H. Miyaki, S. Mihashi and T. Shakunaga, “Photometric
Image-Based Rendering for Image Generation in Arbitrary Illumination,”
Proc. 8th IEEE Int. Conf. on Computer Vision, pp. 643-649, July 2001.
[MTY01a] T. Machida, H. Takemura and N. Yokoya, “Dense Estimation of Surface
Reflectance Parameters by Selecting Optimum Illumination Conditions,”
Proc. 14th Int. Conf. on Vision Interface (VI2001), pp. 244-251, June
2001.
[MTY01b] T. Machida, H. Takemura and N. Yokoya, “Dense Estimation of Surface Reflectance Parameters from Registered Range and Color Images by
Determining Illumination Conditions,” Proc. SPIE Electronic Imaging,pp. 127-134, January 2001.
[MTY02]T. Machida, H. Takemura and N. Yokoya, “Dense Estimation of Surface Reflectance Properties for Merging Virtualized Objects into Real Images,” Proc. 5th Asian Conf. on Computer Vision (ACCV2001), pp. 688693, January 2002.
96

[MY02a]T. Machida and N. Yokoya, “Dense Estimation of Surface Reflectance
Properties of Objects with Interreflections,” Proc. 16th IEEE Int. Conf.
on Pattern Recognition (ICPR2002), pp. 348-351, August 2002.

[MY02b]T. Machida and N. Yokoya, “Virtualizing Real Objects with Surface Interreflections,” Proc. IEEE Int. Conf. on Multimedia and Expo (ICME2002),
pp. 833-836, August 2002.

[MYT03]T. Machida, N. Yokoya and H. Takemura, “Surface Reflectance Modeling of Real Objects with Interrefletions,” Proc. 9th IEEE Int. Conf. on
Computer Vision (ICCV2003), pp. 170-177, October 2003.

[PBMH02] T. J. Purcell, I. Buck, W. R. Mark and P. Hanrahan, “Ray Tracing on
Programmable Graphics Hardware,” ACM Trans. on Graphics, pp. 703712, 2002.
[PDC 03] T. J. Purcell, C. Donner, M. Cammarano, H. W. Jensen and P. Hanrahan,“Photon Mapping on Programmable Graphics Hardware,” Proc. Graphics
Hardware, pp. 265-272, 2003.
[Pen82]A. P. Pentland, “Finding the Illuminant Direction,” Journal of the OpticalSociety of America, Vol. 72, No. 4, pp. 448-455, 1982.

[RH01a]R. Ramamoorthi and P. Hanrahan, “A Signal Processing Framework for
Inverse Rendering,” Proc. ACM SIGGRAPH ’01, pp. 117-128, 2001.

[RH01b]R. Ramamoorthi and P. Hanrahan, “On the Relationship Between Radiance and Irradiance: Determining the Illumination from Images of aConvex Lambertian Object,” Journal of the Optical Society of America,
Vol. 18, No. 10, pp. 2448-2459, 2001.

[RH02]

R. Ramamoorthi and P. Hanrahan, “Frequency Space Environment Map
Rendering,” Proc. ACM SIGGRAPH ’02, pp. 517-526, 2002.

[RPV93]

H. Rushmeier, C. Patterson and A. Veerasamy, “Geometric Simplification for Indirect Illumination Calculations,” Proc. Int. Conf. on Graphics
Interface (GI1993), pp. 227-236, 1993.
97

[SAW91]

F. X. Sillion, J. R. Arvo and S. H. Westin, “A Global Illumination Solution for General Reflectance Distributions,” Proc. ACM SIGGRAPH ’91,
pp. 187-196, 1991.

[SCH93]S. J. Gortlerand P. Schroder, M. F. Cohen and P. Hanrahan, “WaveletRadiosity,” Proc. ACM SIGGRAPH ’93, pp. 221-230, 1993.

[SH81]

R. Siegel and J. Howell, Thermal Radiation Heat Transfer, 2nd edition,
1981.

[Shi90]P. Shirley, “A Ray Tracing Method for Illumination Calculation in
Diffuse-Specular scene,” Proc. Int. Conf. on Grapics Interface (GI1990),
pp. 205-212, 1990.

[Shi91]

P. Shirley, “Discrepancy as a Quality Measure for Sample Distributions,”
Proc. Eurographics ’91, pp. 183-194, 1991.

[SS97]

R. Szeliski and H. Shum, “Creating Full View Panoramic Image Mosaics
and Environment Maps,” Proc. ACM SIGGRAPH ’97, pp. 251-258, 1997.

[SSI99]I. Sato, Y. Sato and K. Ikeuchi, “Acquiring a Radiance Distribution to
Superimpose Virtual Objects onto a Real Scene,” IEEE Trans. on Visualization and Computer Graphics, Vol. 5, No. 1, pp. 1-12, January 1999.

[SSI03]

I. Sato, Y. Sato and K. Ikeuchi, “Illumination from Shadows,” IEEE
Trans. on Pattern Analysis and Machine Intelligence, Vol. 25, No. 3,
pp. 290-300, 2003.

[SSS00]

M. Stamminger, P. Slusallek and H. P. Seidel, “Density Control for Photon Maps,” Proc. Rendering Techniques ’00, pp. 23-34, 2000.

[SWI97]

Y. Sato, Mark D. Wheeler and K. Ikeuchi, “Object Shape and Reflectance
Modeling from Observation,” Proc. ACM SIGGRAPH ’97, pp. 379-387,
1997.

[TS67]

K. E. Torrance and E. M. Sparrow, “Theory for Off-specular Reflection
from Roughened Surfaces,” Journal of the Optical Society of America,
Vol. 57, No. 9, pp. 1105-1114, 1967.
98

[VG95]

H. Veach and L. J. Guibas, “Optimally Combining Sampling Techniquesfor Monte Carlo Rendering,” Proc. ACM SIGGRAPH ’95, pp. 419-428,
1995.

[VG97]

H. Veach and L. J. Guibas, “Metropolis Light Transport,” Proc. ACM
SIGGRAPH ’97, pp. 65-76, 1997.

[War92]G. J. Ward, “Measuring and Modeling Anisotropic Reflection,” Proc.
ACM SIGGRAPH ’92, pp. 265-272, July 1992.

[WCG87]J. R. Wallace, M. F. Cohen and D. P. Greenberg, “A Two-pass Solution
to the Rendering Equation: A Synthesis of Ray Tracing and Radiosity
Methods,” Proc. ACM SIGGRAPH ’87, pp. 311-320, 1987.

[WEH89]J. R. Wallace, K. A. Elmquist and E. A. Haines, “A Ray Tracing Algorithm for Progressive Radiosity,” Proc. ACM SIGGRAPH ’89, pp. 315324, 1989.

[WF91]

P. Whaite and F. P. Ferrie, “Automonous Exploration: Driven by Uncertainty,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.
13, No. 7, pp. 635-657, 1991.

[YDMH99] Y. Yu, P. E. Debevec, J. Malik and T. Hawkins, “Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs,”
Proc. ACM SIGGRAPH ’99, pp. 215-227, August 1999.
[YL89]

N. Yokoya and M. D. Levine, “Range Image Segmentation Based on Differential Geometry: A Hybrid Approach,” IEEE Trans. Pattern Analysis
and Machine Intelligence, Vol. 11, No. 6, pp. 643-649, June 1989.

[ZS95]

K. Zimmerman and P. Shirley, “A Two-pass Realistic Image Synthesis
Method for Complex Scenes,” Proc. Eurographics Rendering Workshop,
pp. 284-295, 1995.

99

List of Publications
Journal Papers
1. T. Machida, H. Takemura and N. Yokoya, ”Dense Estimation of Surface Reflectance Property by Combining Multiple Illumination Conditions”, Transactions of the Institute of Electronics, Information and Communication Engineersof Japan, Vol. J84-D-II, No. 8, pp. 1873-1881, Aug. 2001 (in Japanese). (Chapters 2 and 3).
2. B. Okumura, T. Machida and N. Yokoya, ”Fast Rendering of Reflections in Nonuniform Surfaces Using a Multi-scale Environment Map”, Transaction of theInstitute of Image Information and Television Engineers of Japan, Vol. 57, No.
10, pp. 1300-1306, Oct. 2003 (in Japanese).
3. H. Fukutomi, T. Machida and N. Yokoya, ”Virtual Relighting of a Virtualized
Scene by Estimating Surface Reflectance Properties”, Transaction of the Institute of Image Information and Television Engineers of Japan, Vol. 57, No. 11,
pp. 1576-1582, Nov. 2003 (in Japanese). (Chapter 4).
4. K. Yoshikawa, T. Machida, K. Kiyokawa and H. Takemura, ”A High Presence
Shared Space Communication System Using 2D Background and 3D Avatar”,
Transactions of the Institute of Electronics, Information and Communication Engineers, Vol. E87-D, No. 12, pp. 2532-2539, Dec. 2004.
5. T. Machida, N. Yokoya and H. Takemura, ”Surface Reflectance Modeling ofReal Objects with Interreflections for Augmented Virtuality”, Color Researchand Application (accepted subject to minor revisions). (Chapters 2 and 4).
101

6. T. Machida, H. Takemura and N. Yokoya, ”Inverse Reflectometry for Real Objects with Diffuse and Specular Interreflections”, Transactions of the Institute ofElectronics, Information and Communication Engineers of Japan (in Japanese).
(accepted subject to minor revisions). (Chapter 5).
7. K. Uratani, T. Machida, K. Kiyokawa and H. Takemura, ”A Study of Depth
Visualization Techniques for Virtual Annotations in Augmented Reality Environments”, Transaction of the Virtual Reality Society of Japan, (in Japanese).
(submitted).

International Conferences
1. T. Machida, H. Takemura and N. Yokoya, ”Dense Estimation of Surface Reflectance Parameters from Registered Range and Color Images by Determining
Illumination Conditions”, Proc. SPIE Electronic Imaging, Vol. 4298, pp. 127134, Jan. 2001 (Chapters 2 and 3).
2. T. Machida, H. Takemura and N. Yokoya, ”Dense Estimation of Surface Reflectance Parameters by Selecting Optimum Illumination Conditions”, Proc. 14th
Int. Conf. on Vision Interface 2001 (VI2001), pp. 244-251, Jun. 2001 (Chapters
2 and 3).
3. T. Machida and N. Yokoya, ”Dense Estimation of Surface Reflectance Properties
for Merging into Real Images”, Proc. 5th Asian Conf. on Computer Vision 2001
(ACCV2001), Vol. 2, pp. 688-693, Jan. 2002 (Chapters 2 and 3).
4. T. Machida and N. Yokoya, ”Virtualizing real objects with surface interreflections”, Proc. IEEE Int. Conf. on Multimedia and Expo (ICME2002), Vol. 1, pp.
833-836, Aug. 2002 (Chapters 2 and 4).
5. T. Machida and N. Yokoya, ”Dense estimation of surface reflectance propertiesof objects with interreflections”, Proc. 16th IAPR Int. Conf. on Pattern Recognition (ICPR2002), Vol. I, pp. 348-351, Aug. 2002 (Chapters 2 and 4).
6. H. Takemura, M. Maeda, T. Habara, T. Ogawa and T. Machida, ”Indoor Localization using IR Markers for Augmented Reality”, Demonstration Booklet of Int.
102

Workshop on Man-Machine Symbiotic Systems (MMS-WS), p. 1, Nov. 2002.
7. B. Okumura, T. Machida and N. Yokoya, ”Fast Rendering of Reflections in Nonuniform Surfaces Using a Multi-Scale Environment Map”, Proc. ACM SIGGRAPH 2003, Jul. 2003.
8. T. Machida, N. Yokoya and H. Takemura, ”Surface Reflectance Modeling ofReal Objects with Interreflections,” Proc. 9th IEEE Int. Conf. on Computer
Vision 2003 (ICCV2003), Vol. 1, pp. 170-177, Oct. 2003 (Chapters 2 and 4).
9. M. Maeda, T. Ogawa, T. Machida, K. Kiyokawa and H. Takemura, ”Indoor Localization and Navigation using IR Markers for Augmented Reality”, Proc. Int.
Conf. on Human-Computer Interaction (HCI International 2003), pp. 283-284,
Jun. 2003.
10. K. Yoshikawa, T. Machida, K. Kiyokawa and H. Takemura, ”A High Presence
Shared Space Communication System Using 2D Background and 3D Avatar”,
Proc. Int. Symp. on Applications and the Internet (SAINT2004), pp. 50-55, Jan.
2004 [Best Paper Award].
11. Y. Tomozoe, T. Machida, K. Kiyokawa and H. Takemura, ”Unified GestureBased Interaction Techniques for Object Manipulation and Navigation in a LargeScale Virtual Environment”, Proc. IEEE Virtual Reality (IEEE VR 2004), Mar.
2004.
12. Y. Tomozoe, T. Machida, K. Kiyokawa and H. Takemura, ”A Software Approachto Improve VE Interfaces, Unification of Interaction Techniques for Object Manipulation and Navigation”, Proc. IEEE Workshop on Beyond Wand and Glove
Based Interaction, pp. 101-102, Mar. 2004.
13. T. Machida, H. Takemura and N. Yokoya, ”Dense Estimation of Surface Reflectance Properties Based on Inverse Global Illumination Rendering”, Proc.
17th IAPR Int. Conf. on Pattern Recognition (ICPR2004), Vol. I, pp. 348-351,
Aug. 2004 (Chapters 2 and 5).
14. M. Kanbara, R. Tenmoku, T. Ogawa, T. Machida, M. Koeda, Y. Matsumoto,
K. Kiyokawa, H. Takemura, T. Ogasawara and N. Yokoya, ”Nara palace site103

navigator, A wearable tour guide system based on augmented reality”, Proc.
3rd CREST/ISWC Workshop on Advanced Computing and Communicating Techniques for Wearable Information Playing, pp. 7-14, Oct. 2004.
15. K. Nakashima, T. Machida, K. Kiyokawa and H. Takemura, ”A 2D-3D Integrated Environment for Cooperative Work”, Proc. 14th Int. Conf. on Artificial
Reality and Telexistence (ICAT2004), pp. 348-351, Nov. 2004.

Domestic Conferences
1. T. Machida, H. Iwasa, H. Takemura and N. Yokoya, ”Estimation of ReflectanceParameters Using a Rangefinder under Multiple Illumination Conditions”, Proc.
59th Information Processing Society of Japan, General Conference, No. 2ZC-8,
Sep. 1999 (in Japanese). (Chapters 2 and 3).
2. T. Machida, H. Iwasa, H. Takemura and N. Yokoya, ”Estimation of ReflectanceParameters Using Panoramic Range and Intensity Images under Multiple Illumination Conditions”, Technical Report of the Institute of Electronics, Information
and Communication Engineers of Japan, Pattern Recognition and Media Understanding (PRMU), PRMU99-162, Nov. 1999 (in Japanese). (Chapters 2 and3).
3. K. Nakakita, T. Machida, H. Takemura and N. Yokoya, ”Presentation of Video
Avatars Based on Following User’s Position Using Active Cameras”, Technical
Report of the Institute of Electronics, Information and Communication Engineers of Japan, Image Engineering (IE), IE2000-160, Jan. 2000 (in Japanese).
4. T. Machida, H. Iwasa, H. Takemura and N. Yokoya, ”Estimation of Reflectance
Property with Considering the Self-Shadow”, Proc. the 2000 Institute of Electronics, Information and Communication Engineers, General Conference, No.
D-12-176, Mar. 2000 (in Japanese). (Chapters 2 and 3).
5. T. Machida, H. Iwasa, H. Takemura and N. Yokoya, ”Estimation of Surface
Reflectance Property by Combining 3D Rangefinder and Multiple Illumination
Conditions”, Proc. Meeting on Image Recognition and Understanding (MIRU2000),
Vol. I, pp. 113-118, Jul. 2000 (in Japanese). (Chapters 2 and 3).
104

6. T. Machida, H. Takemura and N. Yokoya, ”Estimation of Object Surface Reflectance Properties Using Multiple Illumination Conditions”, Proc. the 2000
Kansai-section Joint Convention of Institutes of Electrical Engineering of Japan,
No. G15-23, Aug. 2000 (in Japanese). (Chapters 2 and 3).
7. K. Nakakita, T. Machida, H. Takemura and N. Yokoya, ”A Video Avatar Presentation for Cooperative-Work in a Shared Virtual Environment”, Proc. the2000 Institute of Electronics, Information and Communication Engineers, Society Conference, No. D-16-7, Sep. 2000 (in Japanese).
8. T. Machida and N. Yokoya, ”Estimation of Object Surface Reflectance Properties with Considering Interreflection Based on Radiosity”, Proc. the 6th Virtual Reality Society of Japan, Annual Conference, pp. 455-456, Sep. 2001 (inJapanese). (Chapters 2 and 4).
9. S. Kawano, T. Machida, K. Yamazawa and N. Yokoya, ”An Interactive Creationof Human Animation by Combining Multiple Motion Data”, Proc. the 6th Virtual Reality Society of Japan, Annual Conference, pp. 13-14, Sep. 2001 (inJapanese).
10. T. Machida and N. Yokoya, ”Estimation of Object Surface Reflectance Properties Considering Interreflections”, Technical Report of the Institute of Electronics, Information and Communication Engineers of Japan, Multimedia andVirtual Environment (MVE), MVE2001-67, Oct. 2001 (in Japanese). (Chapters
2 and 4).
11. T. Machida and N. Yokoya, ”Estimation of Object Surface Reflectance Properties with Interreflections”, Technical Report of the Institute of Electronics, Information and Communication Engineers of Japan, Pattern Recognition and Media
Understanding (PRMU), PRMU2001-225, Feb. 2002 (in Japanese). (Chapters 2
and 4).
12. T. Habara, T. Machida, T. Ogawa and H. Takemura, ”Evaluation of Indoors Position Detection Using Fiducial Markers”, Technical Report of the Institute ofElectronics, Information and Communication Engineers of Japan, Image Engineering (IE), Vol. 102, No. 216, pp. 65-70, Jul. 2002 (in Japanese).
10513. M. Maeda, T. Ogawa, T. Machida and H. Takemura, ”Position Detection for a
Navigation Support by Augmented Reality Using Infrared Identifications”, Technical Report of the Institute of Electronics, Information and Communication Engineers of Japan, Image Engineering (IE), Vol. 102, No. 216, pp. 59-64, Jul.
2002 (in Japanese).
14. B. Okumura, T. Machida and N. Yokoya, ”A Real-Time Rendering Techniquefor Non-uniform Roughness Surfaces Using a Multi-Scale Environment Map”,
Proc. the 7th Virtual Reality Society of Japan, Annual Conference, pp. 381-382,
Sep. 2002 (in Japanese).
15. H. Fukutomi, T. Machida and N. Yokoya, ”Interactive Virtual Relighting of aReal Scene with Diffuse and Specular Surfaces”, Proc. the 7th Virtual Reality
Society of Japan, Annual Conference, pp. 375-376, Sep. 2002 (in Japanese).
(Chapter 4).
16. S. Li, T. Machida and H. Takemura, ”An IBR Method for Novel Viewpoint Image Synthesis Using Fractal Image Coding”, Proc. the 7th Virtual Reality Societyof Japan, Annual Conference, pp. 421-422, Sep. 2002 (in Japanese).
17. T. Habara, T. Machida, K. Kiyokawa and H. Takemura, ”Indoors Position Detection Using Fiducial Marker for Wearable PC”, Proc. the 7th Virtual Reality
Society of Japan, Annual Conference, pp. 335-336, Sep. 2002 (in Japanese).
18. M. Maeda, T. Ogawa, T. Machida and H. Takemura, ”Position Detection for a
Wearable System Using Infrared Identifications”, Proc. the 7th Virtual Reality
Society of Japan, Annual Conference, pp. 333-334, Sep. 2002 (in Japanese).
19. T. Machida and N. Yokoya, ”Estimation of Surface Reflectance Properties with
Interreflections based on Radiosity”, Proc. Forum on Information Technology(FIT2002), Vol. 3, No. I-25, Sep. 2002 (in Japanese). (Chapters 2 and 4).
20. B. Okumura, T. Machida and N. Yokoya, ”A Real-Time Rendering Techniquefor Non-uniform Surfaces Using a Multi-Scale Environment Map of a Real
Scene”, Technical Report of the Institute of Electronics, Information and Communication Engineers of Japan, Pattern Recognition and Media Understanding(PRMU), PRMU2002-137, Dec. 2002 (in Japanese).
10621. H. Fukutomi, T. Machida and N. Yokoya, ”Virtualizing of a Real Scene for Interactive Virtual Relighting”, Technical Report of the Institute of Electronics, Information and Communication Engineers of Japan, Pattern Recognition and Media
Understanding (PRMU), PRMU2002-136, Dec. 2002 (in Japanese). (Chapter
4).
22. K. Uratani, T. Machida, K. Kiyokawa and H. Takemura, ”Displaying Annotations in Consideration of Depth Information”, Proc. the 2003 Institute ofElectronics, Information and Communication Engineers, General Conference,p. 304, Mar. 2003 (in Japanese).
23. K. Nakashima, T. Machida, K. Kiyokawa and H. Takemura, ”A Cooperative
Work Environment with Projection Images to an IllusionHole Mask”, Proc. the2003 Institute of Electronics, Information and Communication Engineers, General Conference, p. 310, Mar. 2003 (in Japanese).
24. T. Habara, T. Machida, K. Kiyokawa and H. Takemura, ”Evaluation of Indoor
Position Detection for Wearable PC Using Fiducial Markers”, Proc. 63th Information Processing Society of Japan, General Conference, Vol. 5, pp. 199-202,
Mar. 2003 (in Japanese).
25. K. Yamamoto, T. Machida, K. Kiyokawa and H. Takemura, ”A Method of Polygon Conversion from Voxel Data”, Proc. the 2003 Institute of Electronics, Information and Communication Engineers, General Conference, p. 302, Mar. 2003
(in Japanese).
26. K. Yoshikawa, T. Machida, K. Kiyokawa and H. Takemura, ”High Presence
Shared Space Communication Using 2D Background and 3D Avatar”, Technical
Report of the Institute of Electronics, Information and Communication Engineers of Japan, Image Engineering (IE), IE2003-24, Jul. 2003 (in Japanese).
27. Y. Tomozoe, T. Machida, K. Kiyokawa and H. Takemura, ”Unified Gesture
Commands for Manipulation of Remote Virtual Objects and Navigation in a
Large Scale Virtual Space”, Technical Report of the Institute of Electronics,
Information and Communication Engineers of Japan, Image Engineering (IE),
IE2003-39, Jul. 2003 (in Japanese).
10728. K. Uratani, T. Machida, K. Kiyokawa and H. Takemura, ”Evaluation for Virtualizing Depth Information of Annotations in Augmented Reality Environment”,
Technical Report of the Institute of Electronics, Information and Communication Engineers of Japan, Image Engineering (IE), IE2003-40, Jul. 2003 (inJapanese).
29. K. Nakashima, T. Machida, K. Kiyokawa and H. Takemura, ”An Integrated 2D
and 3D Environment for Multi-User Cooperative Modeling”, Proc. the 8th Virtual Reality Society of Japan, Annual Conference, pp. 513-514, Sep. 2003 (inJapanese).
30. K. Yoshikawa, T. Machida, K. Kiyokawa, H. Takemura and Hiroyuki Ohno,”High Presence Shared Space Communication Using Low Resolution Rangefinderand High Resolution Camera”, Technical Report of the Institute of Electronics,
Information and Communication Engineers of Japan, Pattern Recognition andMedia Understanding (PRMU), PRMU2003-197, Jan. 2004 (in Japanese).
31. K. Uratani, T. Machida, K. Kiyokawa and H. Takemura, ”Evaluation for Virtualzing Depth Information of Annotations in Augmented Reality Environment”,
Technical Report of the Institute of Electronics, Information and Communication
Engineers of Japan, Intelligent Transport System (ITS), ITS2003-57, Feb. 2004
(in Japanese).
32. Y. Tomozoe, T. Machida, K. Kiyokawa and H. Takemura, ”Unified Gesturebased Interaction Techniques for Manipulation of Remote Virtual Objects andNavigation”, Technical Report of the Institute of Electronics, Information andCommunication Engineers of Japan, Intelligent Transport System (ITS), ITS200355, Feb. 2004 (in Japanese).
33. K. Nakashima, T. Machida, K. Kiyokawa and H. Takemura, ”An Integrated 2D3D Work Environment for Cooperative Modeling”, Technical Report of the Institute of Electronics, Information and Communication Engineers of Japan, Intelligent Transport System (ITS), ITS2003-56, Feb. 2004 (in Japanese).
34. T. Habara, T. Machida, K. Kiyokawa and H. Takemura, ”Wide-Area Indoor Position Detection using Fiducial Markers for Wearable PC”, Technical Report of108

the Institute of Electronics, Information and Communication Engineers of Japan,
Intelligent Transport System (ITS), ITS2003-76, Feb. 2004 (in Japanese).
35. D. Takada, T. Machida, K. Kiyokawa and H. Takemura, ”A Development Toolkit
for a Shared Wearable AR System”, Proc. the 2004 Institute of Electronics,
Information and Communication Engineers, General Conference, p. 345, Mar.
2004 (in Japanese).
36. T. Machida, N. Yokoya and H. Takemura, ”Estimation of Surface Reflectance
Properties of Real Objects with Interreflections for Augmented Virtuality”, Proc.
Meeting on Image Recognition and Understanding (MIRU2004), Vol. I, pp. 113118, Jul. 2004 [Best Paper Award] (in Japanese). (Chapters 2 and 5).
37. D. Takada, K. Uratani, T. Machida, K. Kiyokawa and H. Takemura, ”Construction of a Development Toolkit for a Shared Wearable AR System”, Proc. the 9th
Virtual Reality Society of Japan, Annual Conference, pp. 333-334, Sep. 2004 (inJapanese).
38. S. Li, T. Machida and H. Takemura, ”Object Shape and Reflectance Modelingfrom a Sparse Set of Images”, Technical Report of the Institute of Image Electronics Engineers of Japan, pp. 57-62, Nov. 2004.

Awards
1. Scientific Incentive Award of the 2000 Kansai-section Joint Convention of theInstitutes of Electrical Engineering of Japan, 2001.
2. TELECOM System Technology Award for Student of the Telecommunications
Advancement Foundation, 2002.
3. Best Paper Award of Meeting on Image Recognition and Understanding (MIRU2004),
2004.

109