Virtual Photometric Environment using Projector
Yasuhiro Mukaigawa1

Masashi Nishiyama2

Takeshi Shakunaga

Okayama University, Tsushima naka 3-1-1, Okayama, 700-8530, Japan
mukaigaw@ieee.org
Abstract
In this paper, we introduce the Virtual Photometric Environment which is oneof mixed reality systems. The Virtual Photometric Environment enables arbitrarily control over the lighting directions and reflection properties of objects. The
environment is realized by photometric pattern projection onto screen objects.
Reflection properties of the photometric patterns are acquired from real objects.
As an application, a demonstration system was constructed which reproduces
the reflection properties of Japanese ceramic arts on a white plaster vase withchanging lighting direction.

1

Introduction

Mixed reality is a concept in which virtual information generated by computer graphics
is mixed with real scenes[1]. Since mixed reality systems can be used as man-machineinterface as well as for collaboration among users, a lot of systems have been proposed
for mixing virtual objects with real scenes[2][3].
Head mounted displays (HMD) are often used as equipment for realizing mixed
reality systems. HMD are classified into two types; optical see-through and video seethrough displays. In optical see-through displays, virtual images are optically mixed
with a real scene. Although the real scene is directly observed through a glass, thevirtual image is late superimposed onto the real scene. In video see-through displays,
in contrast, virtual images are mixed with real images captured by camera, and the user
observes the mixed image on a monitor. Although the virtual image and real image
are synchronized, latency exists between the mixed image and head motion. That is,mixed reality systems based on HMD have problems that users feel less realism.
To cope with these problems, new mixed reality systems using a projector have been
proposed[4]. Virtual images generated by a computer are directly projected onto realscenes. By using a projector, users do not have to wear special devices such as HMD.
The problem of delay is solved, because the virtual image is independent of head motion
as long as the projector and real scenes are fixed.
Projectors have the capability of both overlaying virtual objects on real scenes
and of changing the appearance of real objects by projecting appropriate photometric patterns[5][6]. These systems can be used for a variety of applications, such as
lighting design simulations and virtual museums. However, photometric appearancechanges such as reflection properties have not been treated in earnest in projectorbased mixed reality systems. In this paper, we introduce a system called the Virtual
Photometric Environment which enables us to arbitrarily control lighting directions aswell as reflection properties of objects. The environment is realized by photometricpattern projection onto screen objects. Geometric information in the real world and1
2

Current affiliation is University of Tsukuba, JAPANCurrent affiliation is Toshiba Corp., JAPANPhotometric Pattern

Virtual Light

Projector

Computer

Screen object in real world

User

Figure 1: Principle of the Virtual Photometric Environment.

photometric information in the virtual world are mixed in the real world. Because users
can see the real world without any device, the realism can be improved.
This paper is organized as follows: Section 2 describes the principle of the Virtual
Photometric Environment and presents basic experimental results using a prototype
system. Section 3 describes a method for acquiring reflection properties from real
objects to improve realism. Some experimental results using Japanese ceramic arts arepresented in Section 4. Finally, some conclusions are presented in Section 5.

2

Virtual photometric environment

2.1

Principle

The Virtual Photometric Environment is a new concept that handles photometric properties such as lighting directions and surface reflectance in the framework of mixed reality. In the environment, photometric properties can be arbitrarily controlled by mixing
geometric information in the real world and photometric information in the virtualworld. As shown in Fig. 1, this environment is realized by pattern projection onto 3-Dscreen objects. Projected patterns are generated according to virtual light and virtualreflection properties. The screen object is assumed to have a Lambertian white surface,with no ambient light.

2.2

Photometric pattern generation

If the 3-D shape of a screen object is known, photometric patterns can be generated
by usual computer graphics methods. The lighting position and reflection properties ofthe screen object can be controlled arbitrarily. We assume that the screen object has atextureless Lambertian white surface. The 3-D shape is therefore difficult to measurestably by stereo methods. In our system, the 3-D shape of the screen object is measured
using a rangefinder, as shown in Fig. 2. Several slit patterns are projected onto thescreen object and the scene is captured by a camera. The 3-D shape can be easily
measured based on the principles of triangulation[7].
To add virtual specular reflections to a screen object, the viewing position of theuser also needs to be detected, because the power of specular reflections depends onviewing direction. The 3-D position of the user can be measured by tracking a marker
attached to the user using a pair of cameras. To precisely measure head motion, amagnetic field sensor can also be used.

CameraProjector

Figure 2: 3-D shape measurement of the screen object.
Screen object

3CCD camera

Projector

Figure 3: Prototype system.

(a) Changes to virtual light position.

(b) Changes to reflection property.

Figure 4: Examples of the virtual photometric environment.

Note that if we give only Lambertian reflection properties to the screen object, theviewing position of the user does not have to be detected because Lambertian reflection
is independent of viewing direction. This means that a lot of users can simultaneously
see the Virtual Photometric Environment and that latency never occurs because thephotometric pattern does not have to be updated due to movement of users.

2.3

Prototype system

To demonstrate the principles of the Virtual Photometric Environment, a prototype
system was constructed, as shown in Fig. 3. This system consists of a projector
for projecting photometric patterns and a camera for rangefinder. To simplify thesystem, the distance between the screen object and the user is assumed to be constant.
Therefore, the viewing position of the user is roughly detected by single camera.
First, experimental results are presented in which lighting direction and reflection

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5: Results of pattern projection onto a vase. (a): Screen object. (b): Measured
3-D shape. (c) and (d): Appearance changes due to virtual light position. (e) and (f):Appearance changes due to virtual reflection properties.

properties are arbitrarily controlled. In this experiment, a screen object including a cone
and a pyramid made of paper was used. Figure 4(a) shows photometric appearancechanges due to changes in virtual lighting position. While reflection properties andcamera position are fixed, the virtual lighting position is moved. We can see thatappropriate photometric patterns including diffuse reflections, specular reflections andcast shadows are projected according to the virtual lighting position. Figure 4(b) showschanges in reflection properties. The left and right images show results in which virtual
specular reflections are given to the cone and pyramid, respectively.
Next, experimental results for a white plaster vase are shown in Fig. 5. Figure 5(a)and (b) show the vase and its 3-D shape, respectively. The shape is almost correctly
measured except for the back of the vase, where slit patterns cannot be projected.
Figure 5(c) and (d) show results with the virtual light located on the left and rightside, respectively. Reflection properties are set to be Lambertian, and the texture wasmanually drawn. Figure 5(e) and (f) show different virtual reflection properties. Virtual
specular reflections are added to (f), while (e) has only Lambertian reflection properties.
We confirmed that the appearance of objects can be arbitrarily controlled by projecting
virtual photometric patterns.
Screen objects contain regions that photometric patterns cannot be projected, andthus virtual photometric information cannot be given to those region. For the same
reason, pseudo cast shadows are always observed. Although these problems cannot be
solved in the current prototype system using a single projector, they could be solved
by using multiple projectors.

3

Acquisition of reflection properties

3.1

Acquisition from real objects

In the prototype system described in the previous section, reflection properties given to
the screen object were defined manually. The appearance of the screen object therefore
tends to be artificial. However, improving the realism is very important for practical useof the system. For this, reflection properties of real objects are acquired and reproducedon the screen object, as shown in Fig. 6. By changing the real objects from whichreflection properties are acquired, a variety of photometric information can be given to
the screen object. Since reflection properties are acquired from real objects, realism ofMix

Geometric information
Reflection
properties

Photometric information

Figure 6: Acquisition of reflection properties from real objects.

Nincoming

outgoing

θr

θi

φr

φi

(a) BRDF with four parameters
NN
incoming

outgoingθr

θi

incoming

outgoing

θ ’r

θi

φ

(b) BRDF with three parameters (c) BRDF with two parameters
Figure 7: Angle parameters of BRDF.

the system can be improved[8].

3.2Representation by BRDF

To represent reflection properties, several reflection models, such as the Phong model[9]and Torrance-Sparrow model[10], are often used. These models formulate reflection
properties using a small number of parameters, and are suitable for hardware rendering.
However, decision of parameters of these reflection models is often unstable. Perfect
reproduction of the reflection properties of real objects is therefore difficult.
In our method, formula-based reflection models are not used. Instead, real intensities
observed by camera are directly used as values in the reflection function without needingparameter estimation. To represent reflection properties, the bidirectional reflectance
distribution function (BRDF) is used. The BRDF f4 represents the ratio of outgoingradiance in the viewing direction (θr , φr ) to incident irradiance from a lighting direction(θi , φi ). (θr , φr ) and (θi , φi ) are relative angles to the surface normal N , as shown in
Fig. 7(a). By assuming that the surface is illuminated by a point light source whose
irradiance is L, the observed radiance i can be expressed by
i = f4 (θi , φi , θr , φr )L.

(1)

If BRDFs could be acquired for each surface point, the photometric pattern to be
projected onto the screen object can be generated easily. Photometric patterns are
generated based on Eq. (1), with viewing direction, virtual lighting direction, andlighting power parameters. That is, texture mapping can be applied even if the 3-D
shape is different between the screen object and the real object from which reflection
properties are acquired.

3.3

Angle parameters of BRDF

To perfectly acquire the BRDF specified by four angle parameters, a complex mechanism is necessary to control the position of both camera and light[11]. Moreover, hugememory is required to store the complete BRDFs for every point on the surface. However, the four angle parameters are often redundant. The number of parameters can be
reduced.
When a surface rotates around the surface normal, the appearance of the center
point of the rotation does not change for most materials. Therefore, the angles φi andφr are thus assumed to be redundant, and the BRDF is defined with three parameters,
θi , θr , and φ. φ denotes the sum of φi and φr , as shown in Fig. 7(b). The BRDF
can be approximated by the function f3 with three angle parameters, and the observed
radiance i is expressed by
i = f3 (θi , θr , φ)L.
(2)
Moreover, the angle parameter φ is also redundant for a lot of materials[12]. The
BRDF is thus approximated by the function f2 with two angle parameters θi and θr0 . θr0
denotes the angle between the surface normal and the mirror direction of the incident
light, as shown in Fig. 7(c). Observed radiance i is then given by
i = f2 (θi , θr0 )L.

(3)

It is desirable for the BRDF to correctly express reflection properties using onlya small amount of memory. In consideration of the tradeoff between the amount of
stored data and the ability for reproducing reflection properties, reflection properties
are acquired by the function f2 in our method.

3.4

Interpolation of BRDF

To acquire BRDFs, both light source and camera need to be rotated around the target
object. However, when the camera is rotated, the correspondence of points between
images needs to be found. This is quite difficult, and so the camera is not rotated. The
camera and target object are fixed, and only the light source is rotated.
However, it is difficult to densely rotate the light around the target object. Figure
8 shows some examples of acquired data. Each image shows a visualized BRDF, which

θi
θ ’r

θi

Diffuse reflection

Figure 8: Examples of acquired reflection properties.

Separateddiffuse reflection

Linear interpolation
along θ ’r

Extrapolation
along θ ’r

Extrapolation
along θ i

Acquired data

Specular reflection

θ ’rIntegration

Separated
specular reflection

Linear interpolation
along θ i

Extrapolation
along θ i

Extrapolation
along θ ’r

Figure 9: Interpolation and extrapolation of BRDF. Diffuse reflection and specularreflection are processed separately.

is specified by two parameters θi and θr0 . In these examples, the light source is rotatedaround the target surface point with changing the height at six levels. White regions
indicate that intensity cannot be acquired. If the sampling is sparse, almost of the datacannot be acquired.
In our method, undefined data are interpolated and extrapolated from the acquired
data. Observed intensity can be expressed as the sum of diffuse reflection and specular reflection[13]. The power of diffuse reflections is assumed to be independent ofviewing direction θr0 and power of specular reflections is assumed to be independentof incident direction θi . Specular reflections are assumed not to be observed for large
angles θr0 . Based on these assumptions, complete BRDFs are obtained by interpolationand extrapolation of the acquired data, as shown in Fig. 9. First, diffuse and specularreflections are separated. Next, diffuse components and specular components are linearly interpolated along θr0 and θi , respectively. Then, the uninterpolated regions are
extrapolated. Finally, the complete BRDFs are obtained by integration of the diffuse
and specular reflections.

4

Experimental results

In this section, we show some experimental results that reflection properties are acquired
from real objects and the properties are reproduced on the screen object. To precisely
control the lighting direction, a lighting control system was constructed, as shown in Fig.
10. This system consists of a camera for capturing images, a projector as a rangefinder,and six lights. The lights are installed on a bar, and the bar rotates around the target
object. The rotation angle is arbitrarily controlled by computer. The 3-D position andpower of the lights are calibrated in advance.
Keeping the camera and the target object fixed, the bar with six lights is rotated by

Turn
table

Light

TargetobjectFigure 10: BRDF acquisition system.

θiθr’

Figure 11: Results of interpolation and extrapolation of reflection properties.

1◦ from –100◦ to +100◦ (0◦ indicates the camera direction). The switches of six lights
are independently controlled to be on or off. A total of 1206 images were then taken ina dark room for each target object.
Figure 12 (a) shows a Japanese ceramic art (Bizen-Hidasuki) from which reflection
propertied are acquired. Figure 11 shows some results of acquiring reflection properties
after interpolation and extrapolation. Figure 12 (b) shows the screen object on whichreflection properties are reproduced. Figure 12 (c) shows the result of reproductionof reflection propertied acquired from the ceramic art. Reflection properties can beseen to have been almost correctly acquired with virtual lighting direction arbitrarily
controlled.
Figure 13 (a) and (b) show more results of the acquisition and reproduction ofreflectance properties. Left images show a real vase from which reflectance properties
are acquired. Right images show the results of reproduction of the reflectance properties
on the screen object. The detailed nuance of the reflections is reproduced on the screen
object.

5

Conclusion

In this paper, we proposed a new mixed reality system called the Virtual Photometric
Environment which mixes geometric information of the real world and photometric
information of the virtual world. The environment is realized by photometric patternprojection onto a screen object. By experiments using a prototype system, latency andgeometric mis-alignment, which are problems in HMD systems, were confirmed to be
overcome using a projector. Virtual lighting position and virtual reflection properties
were confirmed to be arbitrarily controlled in the Virtual Photometric Environment.
To improve the realism of the environment, a method for acquiring reflection properties from real objects was introduced. BRDF was used to represent reflection properties.
BRDF, however, needs a huge amount of memory, as it is specified by four parameters.

(a) Ceramic art(Bizen-Hidasuki)(b) Screen object(plaster)(c) Results of reproduction of the acquiredreflection propertied on the screen object.
Figure 12: Experimental results of acquiring reflection properties from a real object
and reproducting these on a screen object.

(a) Ceramic art (Bizen-Goma)(b) Glossy ceramic art

Figure 13: Acquisition and reproduction of reflectance properties. The left images show
the real vase from which reflectance properties are acquired. The right images show theresults of reproduction of the reflectance properties on the screen object.
To reduce memory consumption, a two parameter function was used as an approximation to the complete BRDF. Experimental results demonstrated that acquired reflection
properties can be reproduced on a screen object with a different shape to the real object
from which reflection properties were acquired. Although the BRDF that we used isan approximation, reproduced reflections were confirmed to be almost correct, giving
improved realism.
The surface of the screen object is assumed to be perfectly visible from the projector.
The back of the screen object, however, cannot actually be projected onto. This problem
can be solved by using multiple projectors. Now, attempts to develop a system so that
objects of complex shapes can be used as screen objects are currently underway.

Acknowledgement
This research was partially supported by the Japan Science and Technology Corporation
under the Ikeuchi CREST project and by the Ministry of Education, Science, Sports
and Culture, Grant-in-Aid for Young Scientists (B), 14780290, 2002.

References[1] Y.Ohta and H.Tamura: “Mixed Reality: Merging Real and Virtual Worlds”, Ohmsha / SpringerVerlag, 1999.
[2] Y.Ohta, Y.Sugaya, H.Igarashi, T.Ohtsuki, and K.Taguchi, “Share-Z: Client/Server Depth Sensing
for See-Through Head-Mounted Displays” Proc. ISMR2001, pp.64-72, 2001.
[3] Y.Mukaigawa, S.Mihashi and T.Shakunaga: “Photometric Image-Based Rendering for Virtual
Lighting Image Synthesis”, Proc. IWAR’99, pp.115-124, 1999.
[4] J.Underkoffler, B.Ullmer and H.Ishii: “Emancipated Pixels: Real-World Graphics In The Luminous Room”, Proc. SIGGRAPH’99, pp.385-392, 1999.
[5] R.Raskar, G.Welch, W.C.Chen: “Table-Top Spatially-Augmented Reality: Bringing PhysicalModels to Life with Projected Imagery”, Proc. IWAR’99, pp.64-71, 1999.
[6] R.Raskar, G.Welch, K.Low and D.Bandyopadhyay, “Shader Lamps: Animating Real Objects
With Image-Based Illumination”, Proc. Eurographics Rendering Workshop (ERGW2001), pp.89102, 2001
[7] K.Sato and S.Inokuchi, “Range-Imaging System Utilizing Nematic Liquid Crystal Mask”, Proc.
ICCV’87, pp.657-661, 1987.
[8] Y.Mukaigawa, H.Miyaki, S.Mihashi and T.Shakunaga: “Photometric Image-Based Rendering for
Image Generation in Arbitrary Illumination”, Proc. ICCV2001, pp.652-659, 2001.
[9] B.T.Phong, “Illumination for Computer Generated Pictures”, Commun. ACM, vol.18, pp.311-317,
1975.
[10] K.E.Torrance, and E.M.Sparrow, “Theory for off-specular reflection from roughened surface”, J.
Opt. Soc. Am, vol.57, pp.1105-1114,1967.
[11] P.Debevec, T.Hawkins, C.Tchou, H.P.Duiker, W.Sarokin, and M.Sagar, “Acquiring the Reflectance Field of a Human Face”, Proc. SIGGRAPH2000, pp.145-156, 2000.
[12] S.R.Marschner, S.H.Westin, E.P.F.Lafortune, K.E.Torrance, and D.P.Greenberg, “Image-Based
BRDF Measurement Including Human Skin”, Proc. 10th Eurographics Workshop on Rendering,pp.139-152, June 1999.
[13] S.Shafer, “Using color to separate reflection components”, Color Research and Applications,Vol.10, pp.210-218, 1985.