Systems and Computers in Japan, Vol. 36, No. 14, 2005
Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J84-D-II, No. 8, August 2001, pp. 1864–1872

Fast Image Synthesis of Virtual Objects in a Real Scene with
Natural Shadings
Imari Sato,1 Morihiro Hayashida,2 Fumiyo Kai,3 Yoichi Sato,4 and Katsushi Ikeuchi5
1

The National Institute of Informatics, Tokyo, 101-8430 Japan

2

Bioinformatics Center, Institute for Chemical Research, Kyoto University, Kyoto, 611-0011 Japan

3

Sekisui Chemical Co., Ltd., Urban Infrastructure & Environmental Products Headquarters, Kyoto, 601-8105 Japan
4

Institute of Industrial Science, The University of Tokyo, Tokyo, 153-8505 Japan

5

Interfaculty Initiative in Information Studies, The University of Tokyo, Tokyo, 113-0033 Japan

Key words: computer graphics; computer vision;
mixed reality; illumination distribution estimation.

SUMMARY
We propose a new method for superimposing synthetic objects with natural shadings and cast shadows ontoa real scene whose illumination condition is dynamicallychanging in this paper. In general, high computational costfor rendering virtual objects with convincing shading andshadows, such as interreflections or shadows under area
light sources, prohibits real-time synthesis of such composite images with superimposed virtual objects. To overcome
this limitation, we take advantage of the linearity of therelationship between brightness changes observed on an
object surface and change of illumination radiance in ascene and introduce an efficient rendering technique basedon a linear combination of pre-rendered reference imagesof the scene. We have successfully tested the proposed
method in a natural illumination condition of an indoor
environment to demonstrate how fast the method could
superimpose a virtual object onto the scene with highly
realistic shadings and shadows. © 2005 Wiley Periodicals,Inc. Syst Comp Jpn, 36(14): 102–111, 2005; Published
online in Wiley InterScience (www.interscience.
wiley.com). DOI 10.1002/scj.10155

1. Introduction
Techniques for merging virtual objects with a realscene attract a great deal of attention in media industries
such as film making, television broadcasting, game making,and advertising. The synthesized world called augmented
reality allows us to see a real scene with superimposed
virtual objects and to handle phenomena not only in the realworld but also in a virtual world, while virtual reality
technologies immerse a user in a fully computer-generated
scene.
To enhance the quality of synthesized images in theaugmented reality, three aspects have to be taken into
account [16, 17]: geometry, illumination, and time. Morespecifically, the virtual object has to be positioned at adesired location in the real scene, and the object must
appear at the correct location in the image (consistency ofgeometry). Also, shading of the virtual object has to matchthat of other objects in the scene, and the virtual object must
cast a correct shadow, such as a shadow whose charac© 2005 Wiley Periodicals, Inc.
102

illumination directions [6]. This method further considereda model of skin reflectance to estimate the appearance ofthe face seen from novel viewing directions and underarbitrary illumination.
In this study, we generalize the approach based on thelinearity of scene radiance with respect to illumination
radiance and present an efficient technique for superimposing synthetic objects with natural shadings and cast shadows onto a real scene whose illumination is dynamicallychanging: we consider that a scene consists of both real
objects and synthetic objects with fixed scene geometry,
and the image of a scene viewed from a fixed viewing point
is synthesized under dynamically changing illumination.

teristics are consistent with those of shadows in the realscene (consistency of illumination). Lastly, motions of virtual objects and real objects have to be correctly coordinated (consistency of time).
Consistency of geometry and consistency of time
have been intensively investigated in previous studies.* In
those studies, 3D position sensors of various modalities areused for measuring the position and orientation of a user’s
viewpoint in real time, so that a virtual object can besuperimposed onto the image that the user is looking at [2,19, 3, 11].
For the goal to achieve consistency of illumination,
several methods have been proposed for rendering virtual
objects superimposed onto real scenes using real illumination distributions of the scenes that are either directly measured [10, 14, 5, 9, 23] or indirectly estimated [22]. Those
methods have been successfully demonstrated to be capableof synthesizing composite images with convincing qualities.
However, all of the previously proposed techniques
suffer from a common difficulty to be used in applications
that require real-time processing. High computational costof rendering virtual objects with convincing shading andshadows, for example, interreflections or shadows underarea light sources, prohibits real-time synthesis of highquality composite images with superimposed virtual objects. In the case where composite images have to begenerated in real time, simple rendering algorithms supported by a commonly available graphics hardware need to
be used, and thus high qualities in the synthesized images
are hardly expected.
Alternative approaches have been proposed for rerendering a scene as a linear combination of a set of prerendered basis images of the scene [7, 15, 8]. These
approaches are based on the linearity of scene radiance withrespect to illumination intensities. Since this linearity holds
for scenes with complex geometry and complex photometric properties such as interreflections between objects andcast shadows, photo-realistic appearance of a scene can besynthesized based on this simple linear combination operation.
Most of the previously proposed methods, however,
have been developed for the task of interactive lighting
design. Therefore, basis lights under which a set of basis
images are rendered are intentionally positioned at thedesired locations, so that a scene under desired lighting
configurations can be efficiently synthesized. Recently, inthe context of synthesizing photo-realistic appearance ofhuman skin, Debevec and colleagues introduced a method
for re-rendering a human face based on a linear combination of face images taken under densely sampled incident

2. Linearity between Scene Radiance andIllumination Irradiance
Let us first describe the linearity of the relationshipbetween change of brightness observed on an object surface
and change of illumination radiance in a scene in thissection.
Consider one case where a light source L1 illuminates
an object [Fig. 1(a)] and another case where another lightsource L2 placed at a different location illuminates the same
object [Fig. 1(b)]. If we let I1 be the image observed in thefirst case and I2 be the image observed in the second casefrom the same viewpoint, a novel image I, for which both
L1 and L2 illuminate the object, can then be synthesized as
the sum of I1 and I2 [Fig. 1(c)]:Fig. 1. Linearity between illumination radiance andscene radiance.

*

For a good survey of augmented reality technologies, see Ref. 1.

103

(1)

(3)

Similarly, we can obtain an image I′, which should beobserved when the radiance values of the light sources L1and L2 change by considering a linear combination of I1 andI2 asThen the illumination distribution is approximated bydiscrete sampling of its radiance over the entire surface ofthe extended light source. This can be considered as representing the illumination distribution of the scene by using
a collection of light sources with an equal solid angle. As aresult, the double integral in Eq. (3) is approximated as(2)
where r1 and r2 are scaling factors representing the changesof the radiance values of L1 and L2, respectively [Fig. 1(d)].
For instance, if the radiance of L1 decreases by half, then r1
is set to 0.5. It should be noted that this linearity holds not
only for direct illumination from lights but also for indirect
illumination such as interreflections between or among
objects.
Here distant illumination of the scene is assumed;
light sources in the scene are sufficiently distant from theobjects, and thus all light sources project parallel rays onto
the object surface. Namely, the distances from the objects
to the light sources are not considered.

(4)
where n is the number of sampling directions, L(θi, φi) is
the illumination radiance coming from the direction (θi, φi),which also includes solid angle ωi for the sampling direction (θi, φi).
For instance, node directions of a geodesic dome canbe used for uniform sampling of the illumination distribution. By using n nodes of a geodesic dome in a northernhemisphere as a sampling direction, the illumination distribution of the scene is approximated as a collection of
directional light sources distributed with an equal solid
angle ωi = 2π/n.
Some of the incoming light at point A is reflected
toward the image plane. As a result, point A becomes asecondary light source with scene radiance. The bidirectional reflectance distribution function (BRDF) f(θi, φi; θe,φe) is defined as the ratio of the radiance of a surface asviewed from the direction (θe, φe) to the irradiance resulting
from illumination from the direction (θi, φi). Thus, bysumming up the product of the BRDF and the illumination
radiance, the scene radiance I(θe, φe) viewed from thedirection (θe, φe) is computed as

2.1. Approximation of illumination
distribution
We generalize the approach based on the linearity ofscene radiance with respect to illumination radiance further
so that synthetic objects can be synthesized with natural
shadings and cast shadows onto a real scene under natural
illumination conditions of a real scene.
In order to take illumination from all directions into
account, let us consider an infinitesimal patch of the extended light source, of a size δθi in polar angle and δφi inazimuth as shown in Fig. 2. Seen from the center point A,
this patch subtends a solid angle δω = sinθiδθiδφi. Let L0(θi,φi) be the illumination radiance per unit solid angle comingfrom the direction (θi, φi); then the radiance from the patch
is L0(θi, φi) sin θiδθiδiφi and the total irradiance of thesurface point A is [12](5)where Ii = f(θi, φi; θe, φe)L(θi, φi) cos θi.
Similar to Eq. (2), we obtain an image I′ which should
be observed when the radiance values of the light sources
Li change by ri as a linear combination of Ii as(6)

3. Overview of Proposed Approach
Taking advantage of this linear relationship between
brightness observed on an object surface and radiancevalues of light sources in a scene, the proposed method
synthesizes a new image for novel lighting conditions as
described in the following steps.

Fig. 2. (a) The direction of incident and emitted lightrays. (b) Infinitesimal patch of an extended light source.

104

Step 4: New images Iog and Isg, which should beobserved under the current illumination condition, are obtained as a linear combination of Oi’s and Si’s with themeasured scaling factors ri’s, respectively (Fig. 3.4).
Step 5: Using Iog and Isg, the virtual object is superimposed onto the image of the scene along with natural
shading and shadows that are consistent with those of real
objects (Fig. 3.5). The ray casing algorithm is imposed here;
if an image pixel corresponds to the virtual object surface,
the color of the corresponding pixels in Iog is assigned as thevalue of the pixel. Otherwise, the effects on the real objects
caused by the virtual object, that is, shadows and secondaryreflection, are added by multiplying the pixel value by theratio of Iog to Isg (Fig. 3.5).
The main advantage of the proposed method is thatcomposite images are synthesized by simple linear operations based on reference images pre-rendered as an off-line
process. From this the quality of output images is notaffected by the real-time processing requirement on-the-fly.
In the following, Section 4 explains how to prepare
the reference images, and Section 5 describes a method formeasuring an illumination distribution of a real scene todetermine the radiance scaling factor ri’s. Then Section 6
explains the entire process of superimposing a virtual object
onto the real scene by using our proposed method.

Step 1: The entire illumination of a scene is approximated as a collection of area sources Li(i = 1, 2, . . . , n)which are equally distributed in the scene. In our current
implementation of the proposed method, a set of triangular
patches of a geodesic dome are used as a collection of area
light sources Li(i = 1, 2, . . . , n) that approximate the entire
illumination distribution (Fig. 3.1).
Step 2: Two images which are referred to as reference
images are rendered under each area light source: one witha virtual object superimposed onto the scene Oi, and theother without the object Si (Fig. 3.2).
Step 3: Scaling factors of the light source radiancevalues ri(i = 1, 2, . . . , n) are measured by using anomnidirectional image of the scene taken by a camera witha fisheye lens (Fig. 3.3).

4. Rendering Reference Images
The image acquisition system used in our experimentis shown in Fig. 4.* A 3CCD color camera (Camera A) is

Fig. 4. Image acquisition system. A 3CCD color
camera is used for taking an image of the scene, andanother color camera with a fisheye lens is used for
capturing illumination of the scene.
*
A similar image acquisition setup was used in Ref. 13 to superimpose
synthetic objects onto a real scene under real-world illumination.

Fig. 3. Basic steps of the proposed method.

105

used for taking an image of a real scene onto which a virtual
object is superimposed. In addition, another color camera
with a fisheye lens (Camera B) is used for taking anomnidirectional image of the scene which is then used for
determining the illumination distribution of the scene.
First, we estimate the camera parameters of Camera
A which will be later used for rendering synthetic images.
This is necessary for making virtual objects appear at adesired position in final composite images, that is, consistency of geometry. In our method, this camera calibration
was done by using Tsai’s camera calibration method [21].
This method is known to be able to estimate camera parameters reliably by taking into account various effects
causing image distortions.
Then reference images are rendered using the rendering software called Radiance, which can accurately simulate reflections of lights based on physical models of lights[24]. Using Radiance, the following two reference images
are rendered off-line for each area light source Li.

5. Measuring Illumination Distribution
Next, we describe how to measure radiance scaling
factors ri which represent an illumination distribution of ascene by using an omnidirectional image taken by a camera
with a fisheye lens shown in Fig. 4.*
The fisheye lens used in our system is designed insuch a way that an incoming ray to the lens is projected onto
a certain location in an omnidirectional image according toa desired projection model. Therefore, for each area lightsource, projection of three vertices of the area light source
patch can be determined based on this projection model ofthe lens. Then the radiance scaling factor for the area lightsource is computed by using the average pixel values inside
the projection of the area light source in the omnidirectional
image.
Note that we also need to determine an additional
scaling factor between a unit radiance value used for rendering the reference images and radiance scaling factor ri
that are measured from the omnidirectional image as described above. In our experiment in Section 7, this scaling
factor is manually specified so that surface color of a virtual
object which is rendered using measured radiance scaling
factors becomes a certain value.
In order to automatically determine this scaling factor, it is necessary to perform a photometric calibration
between unit radiance value used for rendering reference
images and illumination radiance measured from an omnidirectional image.

1. Object images Oi are images of a scene that contains a virtual object [Fig. 5(a)].
2. Surface images Si are images of the same scene
without the virtual object [Fig. 5(b)].
For rendering those reference images, we need toknow geometric and photometric models of the real scene,that is, shape and reflectance properties of real objects neara virtual object. Recovery of the geometric and photometricmodel of a real object is not an easy task, and it has been
intensively studied in the past. Since geometric andphotometric modeling of a real object is not the main focus
of our study, those models used for our proposed method
are manually specified by a user.
In addition to the object image and the surface image,another image which we call the mask image is prepared.
The mask image is used for determining whether a ray
extending from the camera projection center Camera A
through each image pixel intersects a virtual object or a real
object in the scene.

6. Superimposing Virtual Objects onto a
Real Scene
After the radiance scaling factors ri are determined,
the virtual object is superimposed onto the real scene withconsistent shading under the captured illumination condition. Note that each of the three color bands (R, G, B) of acomposite image is separately synthesized using the corresponding band of the reference images and the scaling
factor ri’s. For simplicity of discussion, we do not describe
color bands explicitly in this section.
First, a new object image and a new surface image
that should be observed in the measured illumination condition are synthesized as a linear combination of the corresponding reference images Oi’s and Si’s, respectively. Let
the new object image be Iog and the new surface image be
Isg; then those images are obtained as(7)*
3CCD color camera Victor KYF-57 and fisheye lens (Fit Corporation
FI-19 with field of view of 180°).

Fig. 5. Reference image. (a) Shadow image, (b) surface
image; (c) mask image.

106

experiment, SGI Onyx2 with six CPUs was used for capturing a sequence of input images and for synthesizing final
composite images. For efficient generation of the composite images, the following two strategies were employed.

(8)Then, for each image pixel in the composite image,
we determine whether the pixel corresponds to the virtual
object or to a real object in the scene by using the previously
prepared mask image. If a pixel corresponds to a point on
the virtual object surface, a color to be observed at this pointis extracted from the same pixel coordinate of the Iog andstored in the composite image.
Otherwise, we modify an observed color at the pointon the real object surface, so that effects caused by thevirtual object onto the real object are represented. For thispurpose, the ratio of Iog to Isg is first computed. The ratio
represents how much of the color at the real object surface
should be modified if the virtual object was placed in thescene. Let P be the observed color of the image pixel of
Camera A; then the color P′ that should be observed if there
were a virtual object in the scene is computed by using theratio

1. As described in Section 6, each pixel in the composite image requires a different type of operation, depending on which object surface the pixel corresponds to.
Computational cost of synthesizing each image pixel in thecomposite image is taken into account for distributing
computation evenly among all six CPUs.
2. Because effects due to the light sources with low
radiance values are negligible in the final composite image,
the reference images rendered under light sources whoseradiance values are lower than a prefixed threshold value
are omitted when a linear combination of the reference
images is computed. In this way, we can reduce the numberof reference images used for synthesizing final composite
images, and achieve required processing time by adjusting
the threshold value.

(9)

Several examples of synthesized composite images
are shown in Fig. 6: (a) represents input images, (b) represents synthesized composite images, and (c) represents
omnidirectional images that show changes in the real illumination distribution as time passed. In the composite
images shown in this figure, appearance of the virtual object
blends well into the scene, and the virtual object casts a

7. Experimental Results
We have tested the proposed method by using real
images of an indoor environment as shown in Fig. 4. In thisFig. 6. Synthesized results. (a) Input images; (b) synthetic images; (c) omnidirectional images.

107

Table 1. Processing time due to the number ofreference images

shadow with a soft edge on the grass in the same way as theother objects such as trees in the scene do. Table 1 showschanges in processing time due to the number of reference
images.*Since the composite images are synthesized not byrendering the whole scene over and over but rather basedon the linear combination of pre-rendered reference images,
the processing time for synthesizing the composite images
never affects the achievable image quality. In this example,
we approximated the real illumination distribution by using
400 light sources, and light sources whose radiance isgreater than a threshold value are selected for the rendering
process. In spite of the highly realistic shading and shadowsachieved in the composite image, the virtual object was able
to be superimposed onto the scene at a frame rate of
approximately 3 to 4 frames per second.
In contrast, when light sources were not selectedaccording to the threshold value, it took 1.2 seconds togenerate one synthetic image. Here if we use a fewer
number of light sources that approximate the real-world
illumination distribution in order to speed up the rendering
process, the solid angle of each light source (δω = 2/n)increases and this results in blurring the shading and castshadows of the virtual object, which might differ from thoseof real objects.
In the experimental example mentioned above, thenumber of basis images was significantly reduced based on
the proposed light source selection scheme and a high frame
rate was obtained. However, when the number of basis
images to be used for synthesis processing cannot be effectively reduced by this light source selection, a number ofbasis images need to be used and the synthesis processing
speed must be compromised to some extent. We comparedthis frame rate with that of the conventional rendering
technique by assigning a set of light source radiance fromthe omnidirectional image of the scene. Here all of 400 lightsources are used for rendering the scene. We found that it
took at least 8 minutes to render natural shading of thevirtual object by using the Radiance rendering software on
the same SGI-Onyx2 system. Even when light sources wereselected based on their radiance value, it took at least 5
minutes to render it.
It was apparent from this that by using the proposed
approach, the synthesis processing speed was increased
approximately 400 times when light sources were not selected and approximately 1200 times when light sourceswere selected.† One can say that the proposed technique

No. of reference images
6
25
90
190
400

Processing time (s)
0.08
0.2
0.33
0.55
1.2

generated high-quality synthetic images much faster thanthe conventional technique even when a large number oflight sources were used to approximate the illumination
distribution of a scene and many basis images were used forthe image synthesis process.
In the experiment shown in Fig. 7, the 3D shapes andreflectance properties of the surrounding real objects weremodeled by using a photo-modeling tool called Canoma[19]. In Fig. 7, the white hemisphere placed in the center ofthese images is a synthetic object, and the other blocks arereal objects. In the images synthesized by the proposed
technique, we see that the brightness of the virtual objectchanged in a similar manner as the other blocks due tochanges in the real-world illumination distribution. In the
resulting image on the right, the shadows of the real objects
fall naturally on the virtual object. In the synthetic image
on the left, interreflections are observed between the virtual
object and the cube-shaped block located diagonally infront of the virtual object. A color blending phenomenon inwhich the color of the cube is reflected on the virtual object
is also synthesized correctly.

*

As described before, the number of reference images is equal to thenumber of area light sources that approximate the entire illumination
distribution of the scene.
†
However, this comparison does not take into consideration the off-line
processing time required when generating the base images. The comparison is simply performed in terms of the synthesis processing time.

Fig. 7. The 3D shape and reflection properties of thereal objects were modeled and used for rendering basis
images. The shadows of the real objects fall naturally on
the virtual object, and the interreflections are observed
between the virtual object and the real cube-shaped block.

108

2. Azuma RT, Bishop G. Improving static and dynamic
registration in an optical see-through HMD. Proc
SIGGRAPH 94, p 197–204.
3. Bajura M, Fuchs H, Ohbuchi R. Merging virtual
objects with the real world: Seeing ultrasound imagery within the patient. Proc SIGGRAPH 92, p
203–210.
4. Basri R, Jacobs D. Lambertian reflectance and linear
subspaces. Proc IEEE Int Conf Computer Vision ’01,
p 383–389.
5. Debevec PE. Rendering synthetic objects into realscenes: Bridging traditional and image-based graphics with global illumination and high dynamic range
photography. Proc SIGGRAPH 98, p 189–198.
6. Debevec P, Hawkins T, Tchou C, Duiker H, Sarokin
W, Sagar M. Acquiring the reflectance field of ahuman face. Proc SIGGRAPH ’00, p 145–156.
7. Dobashi Y, Kanade K, Naktani H, Yamashita H. A
quick rendering method using basis function for interactive lighting design. Proc Eurographics ’95, p229–240.
8. Dorsey J, Arvo J, Greenberg D. Interactive design ofcomplex time-dependent lighting. J IEEE Computer
Graphics and Applications 1995;15:26–36.
9. Drettakis G, Robert L, Bougnoux S. Interactive common illumination for computer augmented reality.
Proc 8th Eurographics Workshop on Rendering, p
45–57, 1997.
10. Fournier A, Gunawan A, Romanzin C. Common
illumination between real and computer generated
scenes. Proc Graphics Interface ’93, p 254–262.
11. Fujii H, Kanbara M, Iwasa H, Takemura H, Yokoya
N. A registration method using stereo cameras withan inertial sensor for augmented reality. Tech Rep
IEICE 2000;PRMU99-192.
12. Horn BKP. Robot vision. MIT Press; 1986.
13. Koudelka M, Belhumeur P, Magda S, Kriegman D.
Image-based modeling and rendering of surfaces
with arbitrary BRDFs. Proc IEEE Conf Computer
Vision and Pattern ’01, Vol. 1, p 568–575.
14. Miyama M, Ozawa S, Kondo K. Merging of live
action and CG images using estimation of photographic conditions. 2nd Intelligent Information Media Symposium, p 113–118, 1996.
15. Nameroff J, Simoncelli E, Dorsey J. Efficient re-rendering of naturally illuminated environments. Proc
5th Eurographics Workshop on Rendering, 1994.
16. Ohta Y. Topics and expectations in mixed reality
techniques. Publication of the First VR Society
Mixed Reality Technical Symposium ’97.
17. Tamura H. Mixed reality research project. General
Conference of the IEICE, ISS-1-4, 1997.

8. Discussion
There is one limitation in the approaches based on thelinearity of scene radiance with respect to illumination
radiance. That is, an image of the scene under novel illumination condition can be synthesized accurately unless thisillumination condition lies in the linear subspace spannedby the basis lights under which basis images are rendered.
Therefore, it is essential to provide basis lights and theirassociated weights properly so that images under desirable
illumination conditions can be synthesized.
Previous studies investigated how to choose a set ofbasis lights so that these lights could efficiently represent
specific lighting configurations. For instance, Nameroffand colleagues introduced a set of steerable area lightsources as basis lights to approximate the illumination
effect of daylight [15]. Dobashi and colleagues defined aset of basis lights to represent directionality of spotlights
based on spherical harmonics [7]. Later, Teo and colleagues
introduced a hybrid method for synthesizing illumination
effects from both area sources and directional spotlightsources [20]. In addition, this method adopted a strategy for
reducing the number of basis images based on principal
components analysis. Recently, the effect of defining a setof basis lights in the frequency domain based on sphericalharmonics was demonstrated in Refs. 4 and 18.

9. Conclusions
In this paper, we proposed a new method for superimposing virtual objects with natural shading and castshadows onto a real scene whose illumination condition
was dynamically changing. Based on linearity of the relationship between change of brightness observed on an
object surface and change of illumination radiance values
in a scene, final composite images are synthesized as alinear combination of pre-rendered reference images ofvirtual objects placed in the scene in our proposed method.
The main advantage of our proposed method is that rendering algorithms that are computationally too expensive forreal-time processing can be used for rendering reference
images since the rendering process is done off-line. Thus,
the image quality is not affected by the requirement for
real-time processing, and the proposed method is able tosatisfy both high image quality and real-time processing.

REFERENCES
1. Azuma RT. A survey of augmented reality. Presence:
Teleoperators and Virtual Environments.
1997;6:355–385.

10918. Ramamoorthi R, Hanrahan P. A signal-procession
framework for inverse rendering. Proc SIGGRAPH’01, p 117–128.
19. State A, Hirota G, Chen DT, Garrett WF, Livingston
MA. Superior augmented-reality registration by integrating landmark tracking and magnetic tracking.
Proc SIGGRAPH 96, p 429–438.
20. Teo PC, Simoncelli EP, Heeger DJ. Efficient linear
re-rendering for interactive lighting design. Report
No. STAN-CS-TN-97-60, Department of Computer
Science, Stanford University, 1997.
21. Tsai R. A versatile camera calibration technique forhigh accuracy machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J Robotics
and Automation 1987;3:323–344.
22. Sato I, Sato Y, Ikeuchi K. Illumination from shadows.
IEEE Trans Pattern Anal Mach Intell 2003;25:290–300.
23. Sato I, Sato Y, Ikeuchi K. Acquiring a radiance distribution to superimpose virtual objects onto a realscene. IEEE Trans Visualization and Computer
Graphics 1999;5:1–12.
24. Radiance is Unix freeware for lighting design andrendering developed by the U.S. Department of Energy and the Swiss federal government. http://hobbes.lbl.gov/radiance/

AUTHORS (from left to right)

Imari Sato received her B.S. degree in policy management from Keio University in 1994 and M.S. and Ph.D. degrees ininterdisciplinary information studies from the University of Tokyo in 2002 and 2005. She is an assistant professor at the National
Institute of Informatics. Previously, she was a visiting scholar at the Robotics Institute at Carnegie Mellon University, Pittsburgh,
Pennsylvania. Her research interests include physics-based vision and image-based modeling. She is a research fellow of theJapan Science and Technology Agency.
Morihiro Hayashida received his B.S. and M.S. degrees from the Division of Computer Science in the Department of
Physical Science at the University of Tokyo in 2000 and 2002 and Ph.D. degree from the Graduate School of Informatics, Kyoto
University, in 2005. He is an assistant professor at the Bioinformatics Center, Institute for Chemical Research, Kyoto University.
He is engaged in research on the analysis of gene expression networks.
Fumiyo Kai graduated from the Department of Science, Engineering and Information Science of Saga University in 1992.
From 1992 to 1994, she worked on developing simulations of applications for factory production lines and also production
systems at the Kyoto Technology Center–Factory Automation Technology Promotion Center of Sekisui Chemical Co., Ltd.
From 1994 to 1996, she worked on the development of technology information managing systems and technology informationcollection systems intended for managers at the Kyoto Technology Center and Technology Planning Headquarters of Sekisui
Chemical Co.; and from 1996 to 1998, she was engaged in the development of design systems at the Kyoto Technology
Center–CAE Research Laboratories of the same company. Since 1998, she has been engaged in research on computer graphics
as a member of the Kyoto Research Institute CAE Group of Sekisui Chemical Co., Ltd.

110

AUTHORS (continued) (from left to right)

Yoichi Sato (member) received his B.S. degree in mechanical engineering from the University of Tokyo in 1990 and M.S.
and Ph.D. degrees in robotics from the School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania, in
1993 and 1997. In 1997, he joined the Institute of Industrial Science at the University of Tokyo, where he is currently an associate
professor. His primary research interests are in the fields of computer vision (physics-based vision, image-based modeling),
human–computer interaction (perceptual user interface), and augmented reality. His awards include the Yamashita Commemorative Prize in 2000, the International Conference on Shape Modeling and Applications ’97 Most Valuable Paper Prize, and theAnnual Journal of the Virtual Reality Society of Japan Paper Prize in 2000. He is a member of the Virtual Reality Society ofJapan, the Association for Computing Machinery, and IEEE.
Katsushi Ikeuchi (member) received his B.E. degree in mechanical engineering from Kyoto University in 1973 and Ph.D.
degree in information engineering from the University of Tokyo in 1978. He is a professor at the University of Tokyo’s Instituteof Industrial Science. After working at the Artificial Intelligence Laboratory at Massachusetts Institute of Technology, theElectrotechnical Laboratory of the Ministry of International Trade and Industries, and the School of Computer Science at
Carnegie Mellon University, he joined the University of Tokyo in 1996. He has received various research awards, including theDavid Marr Prize in computational vision in 1990 and IEEE R&A K-S Fu Memorial Best Transaction Paper award in 1998. In
addition, his 1992 paper, “Numerical Shape from Shading and Occluding Boundaries,” was selected as one of the mostinfluential papers to have appeared in the Artificial Intelligence Journal within the past 10 years. He is a fellow of IEEE.

111