Proceedings 2003 IEEE International Symposium on
Computational Intelligence in Robotics and Automation
July 16-20, 2003, Kobe, Japan

Realization of Virtual Photometric Environment
by Photometric Pattern Projection
Yasuhiro Mukaigawa
Institute of Engineering
Mechanics and Systems
University of Tsukuba
mukaigaw@ieee.org

Abstract

Photometric Pattern

In this paper, we introduce the Virtual Photometric Environment which is one of mixed reality systems. The
Virtual Photometric Environment enables arbitrarily
control over the lighting directions and reflection properties of objects. The environment is realized by photometric pattern projection onto screen objects. Reflection properties of the photometric patterns are acquired
from real objects. As an application, a demonstration
system was constructed which reproduces the reflection
properties of Japanese ceramic arts on a white plastervase with changing lighting direction.

1

Takeshi Shakunaga
Dept. of Information Technology
Faculty of Engineering
Okayama University

Masashi Nishiyama
Dept. of Information Technology
Faculty of Engineering
Okayama University

Projector

Computer

Screen object in real world

User

Figure 1: Principle of the Virtual Photometric Environment.

Introduction

Mixed reality is a concept in which virtual information generated by computer graphics is mixed with realscenes[1]. Since mixed reality systems can be usedas man-machine interface as well as for collaboration
among users, a lot of systems have been proposed for
mixing virtual objects with real scenes[2][3].
Head mounted displays (HMD) are often used as
equipment for realizing mixed reality systems. HMD areclassified into two types; optical see-through and video
see-through displays. In optical see-through displays,
virtual images are optically mixed with a real scene.
Although the real scene is directly observed through a
glass, the virtual image is late superimposed onto thereal scene. In video see-through displays, in contrast,
virtual images are mixed with real images captured bycamera, and the user observes the mixed image on amonitor. Although the virtual image and real image aresynchronized, latency exists between the mixed image
and head motion. That is, mixed reality systems basedon HMD have problems that users feel less realism.
To cope with these problems, new mixed reality systems using a projector have been proposed[4]. Virtual
images generated by a computer are directly projected
onto real scenes. By using a projector, users do not haveto wear special devices such as HMD. The problem ofdelay is solved, because the virtual image is independentof head motion as long as the projector and real scenes
are fixed.
Projectors have the capability of both overlaying virtual objects on real scenes and of changing the appearance of real objects by projecting appropriate photometric patterns[5][6]. These systems can be used for avariety of applications, such as lighting design simulations and virtual museums. However, photometric appearance changes such as reflection properties have not

0-7803-7866-0/03/$17.00 ©2003 IEEE

Virtual Light

been treated in earnest in projector-based mixed reality
systems. In this paper, we introduce a system called
the Virtual Photometric Environment which enables us
to arbitrarily control lighting directions as well as reflection properties of objects. The environment is realizedby photometric pattern projection onto screen objects.
Geometric information in the real world and photometric information in the virtual world are mixed in the realworld. Because users can see the real world without any
device, the realism can be improved.
This paper is organized as follows: Section 2 describes the principle of the Virtual Photometric Environment and presents basic experimental results using a
prototype system. Section 3 describes a method for acquiring reflection properties from real objects to improverealism. Some experimental results using Japanese ceramic arts are presented in Section 4. Finally, someconclusions are presented in Section 5.

2
2.1

Virtual photometric environment
Principle

The Virtual Photometric Environment is a new concept that handles photometric properties such as lighting directions and surface reflectance in the frameworkof mixed reality. In the environment, photometric properties can be arbitrarily controlled by mixing geometric
information in the real world and photometric information in the virtual world. As shown in Fig. 1, thisenvironment is realized by pattern projection onto 3-Dscreen objects. Projected patterns are generated according to virtual light and virtual reflection properties. The
screen object is assumed to have a Lambertian whitesurface, with no ambient light.

435

CameraProjector(a) Changes to virtual light position.

Figure 2: 3-D shape measurement of the screen object.

Screen object

3CCD camera(b) Changes to reflection property.

(c) Changes to viewing position.
Figure 4: Examples of the virtual photometric environment.

Projector

2.3

Figure 3: Prototype system.

2.2

Prototype system

To demonstrate the principles of the Virtual Photometric Environment, a prototype system was constructed,
as shown in Fig. 3. This system consists of a projector
for projecting photometric patterns and a camera for
rangefinder. To simplify the system, the distance between the screen object and the user is assumed to beconstant. Therefore, the viewing position of the user isroughly detected by single camera.
First, experimental results are presented in whichlighting direction and reflection properties are arbitrarily controlled. In this experiment, a screen object including a cone and a pyramid made of paper was used.
Figure 4(a) shows photometric appearance changes due
to changes in virtual lighting position. While reflection
properties and camera position are fixed, the virtual
lighting position is moved. We can see that appropriate
photometric patterns including diffuse reflections, specular reflections and cast shadows are projected according to the virtual lighting position. Figure 4(b) showschanges in reflection properties. The left and right images show results in which virtual specular reflections
are given to the cone and pyramid, respectively. Figure
4(c) shows results of viewing position changes. We cansee that appropriate specular reflections are projectedaccording to viewing direction.
Next, experimental results for a white plaster vaseare shown in Fig. 5. Figure 5(a) and (b) show the vaseand its 3-D shape, respectively. The shape is almost correctly measured except for the back of the vase, whereslit patterns cannot be projected. Figure 5(c) and (d)show results with the virtual light located on the left
and right side, respectively. Reflection properties are
set to be Lambertian, and the texture was manuallydrawn. Figure 5(e) and (f) show different virtual reflec-

Photometric pattern generation

If the 3-D shape of a screen object is known, photometricpatterns can be generated by usual computer graphics
methods. The lighting position and reflection propertiesof the screen object can be controlled arbitrarily. We
assume that the screen object has a textureless Lambertian white surface. The 3-D shape is therefore difficult
to measure stably by stereo methods. In our system,the 3-D shape of the screen object is measured using a
rangefinder, as shown in Fig. 2. Several slit patterns
are projected onto the screen object and the scene iscaptured by a camera. The 3-D shape can be easily
measured based on the principles of triangulation[7].
To add virtual specular reflections to a screen object, the viewing position of the user also needs to be
detected, because the power of specular reflections depends on viewing direction. The 3-D position of the user
can be measured by tracking a marker attached to theuser using a pair of cameras. To precisely measure headmotion, a magnetic field sensor can also be used.
Note that if we give only Lambertian reflection properties to the screen object, the viewing position of theuser does not have to be detected because Lambertian reflection is independent of viewing direction. Thismeans that a lot of users can simultaneously see the Virtual Photometric Environment and that latency neveroccurs because the photometric pattern does not haveto be updated due to movement of users.

436

Mix

Geometric information(a)

Reflection
properties(b)

Photometric information

Figure 6: Acquisition of reflection properties from real
objects.

(c)

(d)

3

Acquisition of reflection properties

3.1

Acquisition from real objects

In the prototype system described in the previous section, reflection properties given to the screen object were
defined manually. The appearance of the screen object
therefore tends to be artificial. However, improving therealism is very important for practical use of the system.
For this, reflection properties of real objects are acquired
and reproduced on the screen object, as shown in Fig.
6. By changing the real objects from which reflection
properties are acquired, a variety of photometric information can be given to the screen object. Since reflection properties are acquired from real objects, realismof the system can be improved[8].

3.2
(e)

Representation by BRDF

To represent reflection properties, several reflection models, such as the Phong model[9] and Torrance-Sparrowmodel[10], are often used. These models formulate reflection properties using a small number of parameters,
and are suitable for hardware rendering. However, decision of parameters of these reflection models is often
unstable. Perfect reproduction of the reflection properties of real objects is therefore difficult.
In our method, formula-based reflection models are
not used. Instead, real intensities observed by camera are directly used as values in the reflection function
without needing parameter estimation. To represent reflection properties, the bidirectional reflectance distribution function (BRDF) is used. The BRDF f4 represents
the ratio of outgoing radiance in the viewing direction(θr , φr ) to incident irradiance from a lighting direction(θi , φi ). (θr , φr ) and (θi , φi ) are relative angles to thesurface normal N , as shown in Fig. 7(a). By assumingthat the surface is illuminated by a point light source
whose irradiance is L, the observed radiance i can be
expressed by(f)

Figure 5: Results of pattern projection onto a vase. (a):Screen object. (b): Measured 3-D shape. (c) and (d):Appearance changes due to virtual light position. (e)and (f): Appearance changes due to virtual reflection
properties.

tion properties. Virtual specular reflections are addedto (f), while (e) has only Lambertian reflection properties. We confirmed that the appearance of objects canbe arbitrarily controlled by projecting virtual photometric patterns.
Screen objects contain regions that photometric patterns cannot be projected, and thus virtual photometric
information cannot be given to those region. For thesame reason, pseudo cast shadows are always observed.
Although these problems cannot be solved in the current
prototype system using a single projector, they could be
solved by using multiple projectors.

i = f4 (θi , φi , θr , φr )L.

437

(1)

N

N
incoming

outgoingN
incoming

outgoing

θr

θi

θr

φr

φi

φ

(a) BRDF with four parameters

incoming

outgoingθ ’rθi(b) BRDF with three parameters

θi(c) BRDF with two parameters

Figure 7: Angle parameters of BRDF.

θi

If BRDFs could be acquired for each surface point,
the photometric pattern to be projected onto the screen
object can be generated easily. Photometric patterns are
generated based on Eq. (1), with viewing direction, virtual lighting direction, and lighting power parameters.
That is, texture mapping can be applied even if the 3-D
shape is different between the screen object and the real
object from which reflection properties are acquired.

3.3

θ ’r

Figure 8: Examples of acquired reflection properties.

Angle parameters of BRDF

To perfectly acquire the BRDF specified by four angle
parameters, a complex mechanism is necessary to control the position of both camera and light[11]. Moreover,huge memory is required to store the complete BRDFs
for every point on the surface. However, the four angle
parameters are often redundant. The number of parameters can be reduced.
When a surface rotates around the surface normal,
the appearance of the center point of the rotation does
not change for most materials. Therefore, the angles
φi and φr are thus assumed to be redundant, and the
BRDF is defined with three parameters, θi , θr , and φ.
φ denotes the sum of φi and φr , as shown in Fig. 7(b).
The BRDF can be approximated by the function f3 withthree angle parameters, and the observed radiance i isexpressed by
i = f3 (θi , θr , φ)L.
(2)

However, it is difficult to densely rotate the lightaround the target object. Figure 8 shows some examples of acquired data. Each image shows a visualized
BRDF, which is specified by two parameters θi and θr0 .
In these examples, the light source is rotated aroundthe target surface point with changing the height at six
levels. White regions indicate that intensity cannot be
acquired. If the sampling is sparse, almost of the datacannot be acquired.
In our method, undefined data are interpolated andextrapolated from the acquired data. Observed intensity can be expressed as the sum of diffuse reflection andspecular reflection[13]. The power of diffuse reflections
is assumed to be independent of viewing direction θr0
and power of specular reflections is assumed to be independent of incident direction θi . Specular reflections are
assumed not to be observed for large angles θr0 . Basedon these assumptions, complete BRDFs are obtained byinterpolation and extrapolation of the acquired data, as
shown in Fig. 9. First, diffuse and specular reflections
are separated. Next, diffuse components and specularcomponents are linearly interpolated along θr0 and θi ,respectively. Then, the uninterpolated regions are extrapolated. Finally, the complete BRDFs are obtainedby integration of the diffuse and specular reflections.

Moreover, the angle parameter φ is also redundant
for a lot of materials[12]. The BRDF is thus approximated by the function f2 with two angle parameters θiand θr0 . θr0 denotes the angle between the surface normal
and the mirror direction of the incident light, as shownin Fig. 7(c). Observed radiance i is then given by
i = f2 (θi , θr0 )L.

(3)

It is desirable for the BRDF to correctly express reflection properties using only a small amount of memory. In consideration of the tradeoff between the amountof stored data and the ability for reproducing reflection properties, reflection properties are acquired by thefunction f2 in our method.

3.4

4

Experimental results

In this section, we show some experimental results thatreflection properties are acquired from real objects andthe properties are reproduced on the screen object. To
precisely control the lighting direction, a lighting control system was constructed, as shown in Fig. 10. Thissystem consists of a camera for capturing images, a projector as a rangefinder, and six lights. The lights are
installed on a bar, and the bar rotates around the target object. The rotation angle is arbitrarily controlled
by computer. The 3-D position and power of the lights
are calibrated in advance.
Keeping the camera and the target object fixed, theInterpolation of BRDF

To acquire BRDFs, both light source and camera need to
be rotated around the target object. However, when thecamera is rotated, the correspondence of points between
images needs to be found. This is quite difficult, and so
the camera is not rotated. The camera and target object
are fixed, and only the light source is rotated.

438

Diffuse reflection

θiSeparateddiffuse reflection

Linear interpolation
along θ ’r

Extrapolation
along θ ’r

Extrapolation
along θ i

Acquired data

Specular reflection

θ ’rIntegration

Separated
specular reflection

Linear interpolation
along θ i

Extrapolation
along θ i

Extrapolation
along θ ’r

Figure 9: Interpolation and extrapolation of BRDF. Diffuse reflection and specular reflection are processed separately.

θi

Turn
table

Light

θr’

Figure 11: Results of interpolation and extrapolation ofreflection properties.

Targetobjectonto a screen object. By experiments using a prototype
system, latency and geometric mis-alignment, which areproblems in HMD systems, were confirmed to be overcome using a projector. Virtual lighting position andvirtual reflection properties were confirmed to be arbitrarily controlled in the Virtual Photometric Environment.
To improve the realism of the environment, a method
for acquiring reflection properties from real objects wasintroduced. BRDF was used to represent reflection
properties. BRDF, however, needs a huge amount ofmemory, as it is specified by four parameters. To reduce
memory consumption, a two parameter function wasused as an approximation to the complete BRDF. Experimental results demonstrated that acquired reflection
properties can be reproduced on a screen object with adifferent shape to the real object from which reflection
properties were acquired. Although the BRDF that weused is an approximation, reproduced reflections wereconfirmed to be almost correct, giving improved realism.
The surface of the screen object is assumed to be
perfectly visible from the projector. The back of thescreen object, however, cannot actually be projected
onto. This problem can be solved by using multiple
projectors. Now, attempts to develop a system so that
objects of complex shapes can be used as screen objects
are currently underway.

Figure 10: BRDF acquisition system.

bar with six lights is rotated by 1◦ from –100◦ to +100◦(0◦ indicates the camera direction). The switches of six
lights are independently controlled to be on or off. A
total of 1206 images were then taken in a dark room foreach target object.
Figure 12 (a) shows a Japanese ceramic art (BizenHidasuki) from which reflection propertied are acquired.
Figure 11 shows some results of acquiring reflection
properties after interpolation and extrapolation. Figure
12 (b) shows the screen object on which reflection properties are reproduced. Figure 12 (c) shows the result ofreproduction of reflection propertied acquired from theceramic art. Reflection properties can be seen to havebeen almost correctly acquired with virtual lighting direction arbitrarily controlled.
Figure 13 (a) and (b) show more results of the acquisition and reproduction of reflectance properties. Left
images show a real vase from which reflectance properties are acquired. Right images show the results ofreproduction of the reflectance properties on the screen
object. The detailed nuance of the reflections is reproduced on the screen object.

5

Acknowledgement

Conclusion

This research was partially supported by the Japan
Science and Technology Corporation under the Ikeuchi
CREST project and by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Young Scientists (B), 14780290, 2002.

In this paper, we proposed a new mixed reality system called the Virtual Photometric Environment whichmixes geometric information of the real world and photometric information of the virtual world. The environment is realized by photometric pattern projection

439

(a) Ceramic art(Bizen-Hidasuki)(b) Screen object(plaster)(a) Ceramic art (Bizen-Goma)(c) Results of reproduction of the acquiredreflection propertied on the screen object.

(b) Glossy ceramic art

Figure 12: Experimental results of acquiring reflection
properties from a real object and reproducting these ona screen object.

Figure 13: Acquisition and reproduction of reflectanceproperties. The left images show the real vase fromwhich reflectance properties are acquired. The right images show the results of reproduction of the reflectanceproperties on the screen object.

References[7] K.Sato and S.Inokuchi, “Range-Imaging System
Utilizing Nematic Liquid Crystal Mask”, Proc.
ICCV’87, pp.657-661, 1987.

[1] Y.Ohta and H.Tamura: “Mixed Reality: Merging Real and Virtual Worlds”, Ohmsha / SpringerVerlag, 1999.

[8] Y.Mukaigawa,
H.Miyaki,
S.Mihashi
andT.Shakunaga:“Photometric
Image-Based
Rendering for Image Generation in Arbitrary
Illumination”, Proc. ICCV2001, pp.652-659, 2001.

[2] Y.Ohta, Y.Sugaya, H.Igarashi, T.Ohtsuki, andK.Taguchi, “Share-Z: Client/Server Depth Sensing for See-Through Head-Mounted Displays” Proc.
ISMR2001, pp.64-72, 2001.

[9] B.T.Phong, “Illumination for Computer Generated Pictures”, Commun. ACM, vol.18, pp.311-317,
1975.

[3] Y.Mukaigawa, S.Mihashi and T.Shakunaga: “Photometric Image-Based Rendering for Virtual Lighting Image Synthesis”, Proc. IWAR’99, pp.115-124,
1999.

[10] K.E.Torrance, and E.M.Sparrow, “Theory for offspecular reflection from roughened surface”, J. Opt.
Soc. Am, vol.57, pp.1105-1114,1967.

[4] J.Underkoffler, B.Ullmer and H.Ishii: “Emancipated Pixels: Real-World Graphics In The Luminous Room”, Proc. SIGGRAPH’99, pp.385-392,
1999.

[11] P.Debevec, T.Hawkins, C.Tchou, H.P.Duiker,
W.Sarokin, and M.Sagar, “Acquiring the Reflectance Field of a Human Face”, Proc. SIGGRAPH2000, pp.145-156, 2000.

[5] R.Raskar, G.Welch, W.C.Chen:“Table-Top
Spatially-Augmented Reality: Bringing PhysicalModels to Life with Projected Imagery”, Proc.
IWAR’99, pp.64-71, 1999.

[12] S.R.Marschner,
S.H.Westin,
E.P.F.Lafortune,
K.E.Torrance, and D.P.Greenberg, “Image-Based
BRDF Measurement Including Human Skin”,
Proc. 10th Eurographics Workshop on Rendering,pp.139-152, June 1999.

[6] R.Raskar, G.Welch, K.Low and D.Bandyopadhyay,“Shader Lamps: Animating Real Objects With
Image-Based Illumination”, Proc. Eurographics
Rendering Workshop (ERGW2001), pp.89-102,
2001[13] S.Shafer, “Using color to separate reflection components”, Color Research and Applications, Vol.10,
pp.210-218, 1985.

440