Appearance Control in Dynamic Light Environmentswith a Projector-Camera System
Ryo Akiyama∗

Goshiro Yamamoto†Toshiyuki Amano‡

Nara Institute of Science and Technology

Nara Institute of Science and Technology

Wakayama University

Takafumi Taketomi§

Christian Sandor¶

Hirokazu Kato∥Nara Institute of Science and Technology

Nara Institute of Science and Technology

Nara Institute of Science and Technology

A BSTRACT
Appearance of objects change from their original colors according to a lighting condition. Nowadays, a projector-camera system
can act a tool to control colors or textures of objects in each small
area divided in pixel level by light projection onto their physicalsurfaces. Conventional system can change the appearance in realtime under constant environmental lights. However, the changeof environmental lights also influence the appearance. Therefore,
we propose an advanced appearance control method considering
the change of the environmental lights. In particular, we use aprojector-camera system consisting of two cameras and one 3D
projector. In addition, we apply liquid crystal shutter filters in order to estimate reflectances and variations of environmental lights
separately simultaneously in real-time. In this paper, we show ourtheory for the estimation and results under different conditions ofenvironmental lights.
Index Terms: H.5.1 [Information Interfaces and Presentation]:
Multimedia and Informations Systems—Artificial, augmented, andvirtual realities; I.4 [Image Processing and Computer Vision]:
Scene Analysis—Color
1

I NTRODUCTION

Colors and textures are essential elements affecting the appearanceof objects. In general, these elements influence to human perception and cognition visually. In these days, projection mapping technologies are advanced and can control colors and textures on surfaces of physical objects. They have contributed to entertainment
events, museum events, and educational and medical situations, allof which are related to human activities. We believe that improvement of these technologies are necessary studies. In addition, we
need to consider human color perception such as color constancy
for advanced appearance control.
In this work, we focus on an appearance control technology witha projector-camera system, which can project a controlled light ontoa physical surface. The basic technique of appearance control is optical compensation. Nayer et al. [10] and Grossberg et al. [7] proposed an optical compensation method for static situations. Thereafter, Fujii at el. [6] proposed a method for removing appearance
in dynamic situations. Enhancing an original appearance is anothertechnique of appearance control [3], [4] . At initial stage, projection
technologies needed to get the appearance of the object in a white∗ e-mail: akiyama.ryo.ah7@is.naist.jp† e-mail: goshiro@is.naist.jp‡ e-mail: amano@sys.wakayama-u.ac.jp§ e-mail: takafumi-t@is.naist.jp¶ e-mail: sandor@is.naist.jp∥ e-mail: kato@is.naist.jp

light in advance, in other word, they can be used in only static situations. For introducing a feedback structure into appearance enhancement, Amano et al. [1] and Bimber et al.[5] real-time appearance control systems that doesn’t need a pre-capture of objects’ appearance. By applying a control theory, Amano et al. [2] proposeda new method for dynamic appearance control that employed theModel Predictive Control (MPC) algorithm. The introduction of
MPC made the appearance control system robust and easier to findsuitable parameter for the control. However, that feedback systemwith MPC cannot control the appearance correctly when environmental lights are changing since the system regards the environmental lights are constant. The system processes the change of theenvironmental light as the change of reflectances. Moreover, it cannot be applied to most of real situations, in which the environmental
light is changing dynamically.
We propose a method to estimate reflectances of the object surface and variations of environmental lights separately simultaneously in real-time. By estimating them separately, our appearance
control system becomes robust against the change of environmental light. In particular, we develop a projector-camera system thatconsists of one 3D projector, two cameras, and liquid crystal shutter
filters to estimate reflectances and variations of the environmental
light. Liquid crystal shutter filters synchronize to the projector anddivide the view of the projection in a time line. By attaching them
onto each camera lens, one camera can observe only the environmental light, while the other acquires most of all visible lights.
A major contribution of this paper is an algorithm for appearance control that needs no optical calibration in each time whenthe environmental lights change. Even if the environmental lights
changes a little, it might not affect human visual perception so muchdue to color constancy. However, there is a risk that difference between the appearance perceived by humans and the target imageof the projector makes the system unstable when the environmental lights changes enough. While our technique does not measure
human visual perception directly, we think it is necessary to discuss about the perception with our technology in this field. Our
method is still at the first stage, but we believe our method can let
us bring projector-camera systems most of place even with environmental light changes dynamically. For example, we might beable to change colors of objects in outdoor environments. Furthermore, it can be used as a smart illuminator to make it easier to reada newspaper or to see photos anywhere. Our system doesn’t need
any additional calibrations for the change of environmental lighteven when other lightings are switched on or off.
2

R ELATED W ORK

Tsukamoto et al. [11] propose a technique for a cooperative multi
projection system to compensate a color of projections. It is robust
to the change of environmental light, and it can show ideal appearance even when the environmental lights change. However, as an
assumption of technique, the system has a target image in advance,
and the appearance is controlled as almost same to the target image.
It is not particular situation only for this cooperative multi projec-

tion system, but also general radiometric compensation techniques(e.g. [10], [7], and [6]). In other words, this technique doesn’t needto estimate reflectances or environmental light because the systemalready have target images. In contrast, our system doesn’t require
any target image in advance. It needs to estimate the appearance ofthe target objects in a white light first, and generates a target image
with additional image processing. Therefore, their method only canadjust the appearance to a given target image.
Mukaigawa et al. [8] propose a technique of light projection considering BRDF (Bi-directional Reflectance Distribution Function)
to reproduce the appearance of a vase in any lighting situation. To
use BRDF, this system needs to capture images lighted from every
angles in advance. Also, Mukaigawa et al. [9] propose a method
to measure BRDF in high speed. Although can measure BRDF in
shorter time as compared to a previous system, it takes still about
50 seconds. Therefore, it assumes a statical environmental lighting
and it is difficult to perform successive appearance manipulation inreal time. In our system, we assume that a target surface is Lambertian surface, and all of lights reflect like Lambertian reflect in orderto reduce calculation cost.
Amano et al. [2] propose a technique to control an appearance ofa physical surface using a projector-camera system. In thier experiments, saturation enhancement, color removal, color phase control,edge enhancement, making blur, making unique brightness, and soon are shown as possible appearance control representations. These
many representations are not limited to amusement applications,but also support for partially color-blind people. This technique estimates reflectances under an assumption of statical environmental
lighting. Therefore, it is undesirable to be used in situation where
the environmental lights change, such as an illuminated environment, including sunlight, or including cast shadow that made byusers or some moving objects. We aim to make this method muchrobust against the change of the environmental lights by applying
additional estimation of variation of environmental lights.
3

S YSTEM D ESIGN

We aim to estimate reflectances of a target object surface and variations of environment lights simultaneously in real-time. Since it isdifficult for only one camera to estimate both, we use two cameras
in a projector-camera system. This section describes the system
structure, an estimation method, and appearance control algorithm.
3.1 System Structure
Our system consists of a projector and two cameras. The projector
has a 3D function that can switch pairs of images alternatively athigh speed. One of the cameras assigned as the “main camera” andthe other is the “sub camera”. In addition, we use liquid crystal
shutter filters (LCSF) in order to appropriately cut off projection
light for the sub camera. By synchronization between these filters
and to the projector, we can make a structure allowing only the main
camera capture the projected light as shown in Fig. 1.
Let the relations between reflectances and lights in each camerabe
Icm = K(I p + I0 + I f ),Ics = K(I0 + I f ),where Icm ∈ R3 is light received by the main camera, Ics ∈ R3 islight received by the sub camera, I p ∈ R3 is projection light, I0 ∈ R3
is default environmental light, and I f ∈ R3 is variations of environmental light. I p , I0 , I f are mixed and reflected on the surface withreflectances K ∈ R3×3 . By using the Lambertian model, we assume
Icm ≈ Ic′ . In other words, we consider that the captured view by themain camera is close to what we can see. Fig.2 shows the relationships between these lights.

Figure 1: System structure: two alternative images are projectedfrom a 3D projector, and each of those images is captured by a corresponding camera through a liquid crystal shutter filter.

In addition, we need to consider differences of color spaces andview points between each device. It is necessary to know geometrical correspondence in one color space for an estimation method forreflectances and variations of environmental light.
First, we calibrate geometrical correspondences between pixelsof the main and sub cameras. Even if we place the sub camera as
close to the main camera as possible, it cannot be exactly the sameview point unless we use a coaxial optical system. In our case, wetransform the view of the sub camera to the view of the main camera
by using the geometrical correspondences. Pixel mappings between
coordinates of “the main camera and the projector” and “the sub
camera and the projector” are calibrated with a general calibration
method employing gray code pattern projection. From these two
pixel mappings, the other pixel mapping between coordinates ofthe main and sub cameras can be calculated. The view of the sub
camera can be transformed to that of the main camera using the
pixel mapping.
Second, we calculate color mixing matrices since the color ineach device needs to be represented in an unified color space. We
use the color space of the main camera as the unified color space forthe calculation of appearance control. The relationships between
the cameras and a projector can be shown as
Cm = Mmp P +Cm0 ,(1)

Cs = Msp P +Cs0 ,(2)where Mmp is the color mixing matrix from the main camera color
space to the projector color space, Msp is the matrix from the sub
camera to the projector, Cm is the captured image of main camera,
Cs is the captured image of sub camera. In addition, Cm and Cs
consist of the projected image P and the captured image without
projection Cm0 and Cs0 , respectively. In this calibration phase, both
cameras need to capture the reflected light of the projection. The
filter for the sub camera must change the sync in order to obtain the
projected light. This is the reason why the sub camera can take theprojection light P in eq. (2), while the projection light is blocked in
the control phase. After the calibrations for Mmp and Msp , we also
can convert the color of the sub camera into the main camera color
space with using eq.(1) and eq.(2).
−1(Cs −Cs0 ) +Cm0
Cm = Mmp Msp

3.2 Estimation Method of K and F
We use a projector-camera response model. A captured image ofmain camera Cm and a captured image of sub camera Cs are generated from the reflected light of the projections P, light with noprojection Cm0 and Cs0 , and variations of environmental light F andF ′ as follows. We define P and F for the main camera color space,

I0+IfMPC
Controller

P

Projector

Ip-+ R Image

Cest

Processing

Cs

K
Sub
CameraImage
Estimater

Cm

Main
Camerawhere P(t), C(t) ∈ ([0, 1], [0, 1], [0, 1]) are a normalized projection
pattern and a captured image by the main camera at the step t. It
should be noted here that all parameters related to camera in MPC
process are generated by the main camera only and we avoid toput “m” as a symbol for the main camera. Image prediction that
contains model error is written by

Ics

C p (t + 1) = CM (t + 1) + Error(t),(9)

Error(t) = C(t) −CM (t).

(10)

where
Icm
Ic'

And the reference trajectory is

Figure 2: A process flow for appearance control with model predictive
control algorithm.

F ′ for the sub camera color space. In this paper, we use ./ and ⊙ ascomponent-wise division and multiplication, respectively.
Cm = K{(Cm f ull −Cm0 )⊙P +Cm0 + F},
Cs = K(Cs0 + F ′ ).

(3)
(4)

In eq. (4), there is no projected light because of cutting off by LCSF.
Considering color mixing matrices, we can calculate an estimated variations of the environmental lights as follows;F̂ = K −1 MmsCs − MmsCs0 .

(5)

Considering eq. (3) and eq. (5), we can express estimated reflectances K̂ without using F̂.
K̂ = (Cm − MmsCs )./{(Cm f ull −Cm0 )⊙P +Cm0 − MmsCs0 }(6)Equations (5) and (6) show that we can calculate K̂ and F̂ simultaneously. Based on K̂, the true appearance that under the white
image Cwhite = (1, 1, 1)T is given by
Cest = K̂Cwhite ,(7)

without interference from F theoretically.
We can make a reference image R by processing some effect onto
the estimated true appearance Cest . Then, we can calculate the subtraction between Cest and the current appearance C for a negative
feedback loop. However, the error of estimated result Cest affects
creation of R since the relationship Cest and R is sensitive. In otherwords, the error of Cest would also be enhanced during appearance
enhancement, then the feedback may be unstable. For that reason,
we need to consider about the model error and the reference trajectory. In section 3.3, we introduce a model predictive control
algorithm.
3.3 Model Predictive Control
Amano et al. [2] proposed an appearance control with negative
feedback based on Model Predictive Control (MPC) algorithm. The
system has achieved high quality and robustness for appearance
control as compared to conventional methods. This is the reason
why we also apply MPC algorithm to our control function. Figure
2 shows the structure of our system with MPC controller.
To apply MPC algorithm, we use projector-camera responsemodel considering variations of the environmental light F(t) as
CM (t + 1) = K̂{(C f ull −C0 ) ⊙ P(t + 1) +C0 + F(t)}(8)

CR (t + 1) = α C(t) + (1 − α )R(t + 1).

(11)

Based on the control law CP (t + 1) = CR (t + 1) with eq. (8) to eq.
(11), we can get manipulating value as follows;P(t + 1)
= K̂(t)−1 (1 − α ){R(t + 1) −C(t)}./(C f ull −C0 )+ K̂(t)−1 K̂(t − 1){C0 + F(t − 1)} −C0 − F(t)≈ K̂(t)−1 (1 − α ){R(t + 1) −C(t)}./(C f ull −C0 ) + P(t).

(12)

In eq. (12), there are approximations of K̂(t)−1 K̂(t − 1) ≈ 1 andF(t − 1) − F(t) ≈ 0. The equation could be simple, while there is anegative possibility that a system become unstable when K is smallvalue. By comparing an equation in previous study [2] and eq. (12),we can find these two equations are completely same. Therefore,
the MPC algorithm we use is the same one what Amano al et. show
in a reference [2].
4P ROTOTYPE S YSTEM

We explain a structure of our prototype system in this section. We
use Allied Vision PIKE as a main camera and Allied Vision Guppy
PRO as a sub camera. They are located close to a projector (EPSONELPGS03) as shown in Fig 3 (a) in order to use large common spaceas the target area as shown in Fig 3 (b). We implement the prototype
system with these 3 devices.
We use a plane surface of a board board located in front of them,and put a white paper on it as target area. In our assumption, thesurface to be projected is the Lambertian surface. Therefore, we
limit the target object is printed papers. In addition, papers withimages printed by a inkjet printer don’t reflect lights on the surface so much like specular reflection as compared to ones printed
by a laser printer. Also, in order to equalize the distance between
the projector and the board, we install the projector parallel to thefloor, and the board vertically to the projection axis. We preferred
to avoid troubles caused by that the brightness of the projection
has unevenness, while that kind of unbalanceness can be compensated in advance. We use EPSON ELPGS03 which is a pair of 3D
active glasses with liquid crystal shutter filters, and put them onto
each camera lens. We describe each spec of our equipment here;
2.7GHz CPU computer (MacBook Pro Intel Corei7 2.7GHz), C++with libraries of OpenCV, and OpenGL for the implementation of
the system. Table 1 shows a list of main equipment and tools forour implemented system.
In a practical case, we need to set 3D glasses appropriately according to situations. We use a L image channel of the 3D projector
for appearance control by using side-by-side 3D mode. In any case,
we locate the left filter to the main camera, while the sub camera
needs to switch each filter according to the process phases. In thecalibration phase, we must locate the left filter for the sub camera as
well. On the other hand, in the appearance control phase, we need tochange it with the right filter to block the projected light. Therefore,Object Area

Sub
CameraMain Camera

Target Area

FilterFilterProjector

(a)

(b)

Figure 3: An implemented system with two cameras attached filters
and a projector: (a) overview of the system, (b) a scene the projection
light onto the target area controls appearance of the object.

Table 1: Equipment and tools for the implementation.
Main Camera
Allied Vision Pike (640x480)
Sub Camera
Allied Vision Guppy PRO (640x480)
Projector
EPSON EH-TW5200 (1080p)
Liquid Crystal Shutter Filter
EPSON ELPGS03
Computer
MacBook Pro OS X 10.9.2
Intel Core i7 @2.7GHz
Software
C++, OpenCV, OpenGL

the main camera can capture the projection and the environmental
light, and the sub camera only can capture the environment light.
Figure 4 shows an estimated image Cest as shown in eq. (7) under
the appearance control with the gray-scale effect without variationsof environmental light F. This estimation should be the same to themethod of Amano et al. Figure 5 show the results that are estimatedtrue appearance related to K and estimated variations of environmental lights F in different situations. We prepared five different
environmental light changes by using a flashlight; white, red, green,
blue, and yellow. Through the results of (w2), (w5), (r2), (r5), (g2),(g5), (b2), (b5), (y2), (y5), interference onto the estimation of K
seems not to be huge by adding the variations of environmental
light except the situation with red light. Similarly, considering theresults of (w3), (w6), (r3), (r6), (g3), (g6), (b3), (b6), (y3), (y6),the estimation of the variations of environmental light can extract
F except the situation with red light. By looking at histogram data,
sensibility of cameras against red color might be lower as comparedto other color channels.
5

D ISCUSSION

Color constancy is one of human visual perceptions. A human eye
can perceive an original color of an object even in different environmental lights. When we use a projector-camera system forappearance control, global illuminations such as room lights on aceiling don’t influence so much in order for a human eye to perceive the original color. Conversely, there is perceptible significant
differences from original colors in small area such as pixel level. If
the system has a function to estimate only reflectances of objects, itcannot detect the original color of objects in a dynamic light environment since the system regards change of environmental lights aschange of reflectances. For example in case of saturation enhancement process, additional red environmental lights let the system estimate reflectance reddish, then the controlled appearance will bemuch more reddish even if a corresponding object color is blue.
On the other hand, our system can distinguish environmental lightsfrom reflectances. Therefore, our system would be able to provide
a proper appearance for human vision considering color constancy
by preparing reference images with colors of environmental lights.
In addition, the system also can realize the original colors by com-

Figure 4: An estimated image under the condition that there is novariation of environmental light in gray-scalse effect process.

pensating the environmental lights.
A novel point of our system is to estimate variation of environmental lights as the same time to estimate reflectances. As variationof environmental lights, we expect a positive and a negative value
such as additional lighting or shadow by blocking lighting, respectively. In case of the positive, we may make a function to estimate
a light source direction using light gradients. If the system can detect the light direction, the appearance would be controlled suchas additional shadowing on target images. On the other hand, we
can extract shadow on a projection area using the negative variations of environmental lights. By extracting the shadow areas, we
may make functions to diminish or enhance the shadow for some
applications such as art work or novel user interface.
6 C ONCLUSION AND F UTURE W ORK
In this paper, we confirmed that the method to estimate reflectances
K and variations of environmental light, which is F, for the appearance control system using two cameras and one 3D projector.
Through this work, we could show the possibility for improvementof the robustness of the appearance control system even though theenvironmental lights change dynamically. However, the quality of
estimation is not enough high yet. For example, red color channel is still not controlled well as shown in Fig. 5 (r3). In order toapply our system for human visual perceptual experiments or applications, we need to cover a high dynamic range for appearance
control. In addition, attractive applications should be implemented
with this technology in the future.
ACKNOWLEDGEMENTS
This work was supported in part by Research Funds from Foundation from Nara Institute of Science and Technology.
R EFERENCES[1] T. Amano and H. Kato. Real world dynamic appearance enhancementwith procam feedback. In Proceedings of the 5th ACM/IEEE International Workshop on Projector Camera Systems, page 5, 2008.
[2] T. Amano and H. Kato. Appearance control using projection withmodel predictive control. In 20th International Conference on Pattern
Recognition, pages 2832–2835, 2010.
[3] T. Amano and R. Suzuki. Virtual recovery of the deteriorated art object
based on ar technology. In IEEE Conference on Computer Vision and
Pattern Recognition, pages 1–2, 2007.
[4] O. Bimber and D. Iwai. Superimposing dynamic range. In ACM
Transactions on Graphics, volume 27, page 150, 2008.

[5] O. Bimber, D. Klöck, T. Amano, A. Grundhöfer, and D. Kurz. Closedloop feedback illumination for optical inverse tone-mapping in lightmicroscopy. In IEEE Transactions on Visualization and Computer
Graphics, volume 17, pages 857–870, 2011.
[6] K. Fujii, M. D. Grossberg, and S. K. Nayar. A projector-camera system with real-time photometric adaptation for dynamic environments.
In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, pages 814–821, 2005.
[7] M. D. Grossberg, H. Peri, S. K. Nayar, and P. N. Belhumeur. Making one object look like another: Controlling appearance using a
projector-camera system. In Proceedings of the 2004 IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, volume 1, pages 452–459, 2004.
[8] Y. Mukaigawa, M. Nishiyama, and T. Shakunaga. Virtual photometricenvironment using projector. In Proceedings of International Conference on Virtual Systems and Multimedia, pages 544–553, 2004.
[9] Y. Mukaigawa, K. Sumino, and Y. Yagi. Rapid brdf measurement
using an ellipsoidal mirror and a projector. In IPSJ Transactions on
Computer Vision and Applications, volume 1, pages 21–32. Information Processing Society of Japan, 2009.
[10] S. K. Nayar, H. Peri, M. D. Grossberg, and P. N. Belhumeur. A projection system with radiometric compensation for screen imperfections.
In ICCV workshop on projector-camera systems, volume 3, 2003.
[11] J. Tsukamoto, D. Iwai, and K. Kashima. Radiometric compensation
for cooperative distributed multi-projection system through 2-dof distributed control. In IEEE Transactions on Visualization and Computer
Graphics, volume 21, pages 1221–1229, 2015.

Figure 5: Experimental results with five different color lights as additional environmental lights in the following order; white, red, green, blue, and yellow.
first column: scenes that an environmental lights change by a flashlight, second column: estimated true appearance, third column: estimated environmental
light variations [×5], fourth column: captured images of the main camera, fifth column: difference between the images of second column and the image of Fig. 4 [×5],sixth column: histograms of the environmental light that is the photos of third column. White, red, green, blue, and yellow lights are used in each rows.