Light Projection-Induced Illusion for Controlling Object Color
Ryo Akiyama*

Goshiro Yamamoto†Toshiyuki Amano‡

Nara Institute of Science and Technology

Kyoto UniversityWakayama University

Takafumi Taketomi§

Alexander Plopski¶

Nara Institute of Science and Technology

Nara Institute of Science and Technology

Christian Sandor||Hirokazu Kato**

Nara Institute of Science and Technology

Nara Institute of Science and Technology

A BSTRACT
Using projection mapping, we can control the appearance of realworld objects by projecting colored light onto them. Because aprojector can only add illumination to the scene, only a limited color
gamut can be presented through projection mapping. In this paper
we describe how the controllable color gamut can be extended byaccounting for human perception and visual illusions. In particular,
we induce color constancy to control what color space observers
will perceive. In this paper, we explain the concept of our approach,and show first results of our system.
Index Terms: Computing methodologies—Computer graphics—
Graphics systems and interfaces—Mixed / augmented reality; Computing methodologies—Computer graphics—Graphics systems andinterfaces—Perception
1I NTRODUCTION

An object’s appearance is affected by a variety of factors, like reflectance, color, and texture. By modifying these factors throughprojection mapping it is possible to alter the object’s appearance.
Jones et al. [5] utilize multiple projectors and depth cameras to convert a room into a mixed reality environment that can be sharedwith other people. Their system projects light not only onto thewalls of the room, but also the players to create an immersive experience. Similarly, Fujimoto et al. [4] manipulate the appearance of adeformable cloth by projecting the desired pattern onto it to assist
designers in evaluating their work.
Projection systems manipulate appearance by overlaying colored
light onto the surface. The projected light mixes with the surface’s
color and is perceived by an observer. The mixing of the backgroundand projected color is the largest drawback of projection systems,
because the system cannot manipulate the appearance to match acolor that cannot be represented as a mixture of these colors. For
example, a red object cannot be made to appear blue, even if blue
light is projected onto it by a powerful projector because it mainlyreflects red light and absorbs the majority of blue and green light.
In this paper we explore how to extend the color gamut a projector
system can express by considering how humans perceive colors. We
perceive the color of an object as a relative value that is influenced
by the color of its surroundings. Therefore, in some cases there is alarge difference between the perceived and actual color of an object.
* e-mail: akiyama.ryo.ah7@is.naist.jp† e-mail: goshiro@kuhp.kyoto-u.ac.jp‡ e-mail: amano@sys.wakayama-u.ac.jp§ e-mail: takafumi-t@is.naist.jp¶ e-mail: plopski@is.naist.jp|| e-mail: sandor@is.naist.jp** e-mail: kato@is.naist.jp

IEEE Virtual Reality 201818-22 March 2018, Reutlingen, Germany© IEEE 2018

This phenomenon is called color constancy. Our idea is to employ
color constancy to present colors that cannot be generated by naı̈ve
projection mapping.
2R ELATED W ORK

In projection-based research, most studies employ a projectorcamera system to control projection colors and to optimize theappearance of the projection surface. Radiometric compensation
technology can negate the effect of colors or texture on a projectionsurface in order to reproduce as closely as possible the intended
appearance of digital content [3]. In contrast to the compensation,
Amano et al. [2] developed a method to control an object’s appearance by light projection based on its original appearance. This work
utilizes the original appearance of an object to enhance some ofits elements. Akiyama et al. [1] extended their method to estimate
both the object’s reflectance and environmental light simultaneouslyto control the object’s appearance in dynamic lighting conditions.
These methods assume that the colors captured by a camera are identical to those perceived by human. However, this is not necessarily
the case.
Human perception has been exploited by Kawabe et al. [6] tocreate an illusion of motion on a static object. To create this illusion
they project a dynamically modified grayscale image to vary theluminance. This continuous modification is perceived as motion byobservers. Although this work also induces a visual illusion, they
focus on perceptual movement. On the other hand, we focus on themanipulation of the perceived object color. Madi et al. [7] created amodel of color constancy for projection color compensation. They
consider visual color illusion to simulate the appearance of an object
under different conditions. They applied their system to preserve theperceived appearance of images. Contrary to their system, our goal
is to freely control an object’s color by accounting for how the scene
is perceived by a human observer.
3

P ROJECTION METHOD

In this section, we explain the color gamut of a projection mapping
system and how it can be expanded by inducing color constancy.
3.1

Limitation of Simple Overlaying Light Projection

As we mentioned before, the color gamut that can be expressed byprojection mapping is limited, because the projector is adding lightto the scene. The limitation depends on the object’s reflectance,
the environmental light, and the specifications of the projector. In
this section, we explain how we can present colors that are outside
the presentable range. We assume that the scene consists only oflambertian surfaces and is illuminated by white light.
Let CL be the color of the real-world object under white environmental light, and CP the color of the projected light that has
been reflected off the object’s surface. This light can be controlled
by adjusting the color emitted by the projector. Our eyes capture
the summed reflected light C = CL +CP . As shown in Figure 1, bymodifying the projected light, we can control CP . C can appear as

RR’

CT

CLmax: (RL+RPmax,GL+GPmax,BL+BPmax)

T
C’T

C = CL+CP(a)B
B’G’

GFigure 1: Through projection mapping a CL colored object can appear
as any color within the color gamut represented by the colorful cuboid.
It thus cannot appear as CT which lies outside. The axes R, G, and B
represent the brightness of the reflected elements of red, green, andblue, respectively. Color constancy transforms the color space by T.
CT is transformed to CT0 which falls within the color gamut.

any color within the color gamut (represented by the colorful cuboid)
In other words, colors outside of the cuboid cannot be presented inthis condition. If we use a more powerful projector, the cuboid becomes larger. In addition, if environmental light is darker, the cuboid
approaches the origin point. Therefore, the color gamut is mainly
determined by the specifications of the projector and environmental
light.
3.2

MethodIn this section, we explain a scheme for presenting a color thatis outside of the controllable range by inducing color constancy.
Assume that for a user to perceive the color OT , CT must be reflected
off the surface of the object. If the vector towards CT intersects thecuboid, it can be presented by naı̈ve projection. However, if thevector does not intersect the cuboid, as shown in Figure 1, it cannot
be presented. Our method can be applied to manipulate the object’sappearance, so that an observer nonetheless perceives the reflectedcolor as OT .
By color constancy effect, humans perceive colors as relatively
constant even under different uniform environmental light. Thiscan be explained as converting color spaces. In Figure 1, due tocolor constancy the RGB space is converted to R’G’B’ space by theconversion matrix T.
In order to calculate T we need to consider the colors CT and CS ,the color of its surrounding area. CS is determined by the object’s
reflectance KS = diag(ks1 , ks2 , ks3 ) and the environmental light L =(RL , GL , BL )T as CS = KS L. By projecting Ps = (Rs , Gs , Bs )T onto
the surrounding area, CS appears as CS0 . Given the additive featureof projection mapping systems, CS0 can be described as
Cs0 = Ks (Ps + L) = λ TKs L,(1)

where λ = kCs0 k/kCs k. T can thus be recovered from CS0 , CS , KS andL. Although PS is not projected onto CT it appears as
CT0 = TCT(c)Figure 2: Experiment results: (a) We want to change the appearanceof the rose and pink colors to blue and gray, respectively. (b) The
target appearance cannot be achieved by simple projection mapping.
(c) By inducing color constancy we achieve the target appearance.

CL: (RL,GL,BL)

O(b)

(2)

due to color constancy. Our goal is thus to find a suitable PS so thatthe resulting color CT0 falls within the color gamut of our system.
We show an example of our projection system being used in
Figure 2. Our goal is to modify the appearance of two colors (highlighted by blue rectangles) on a ColorChecker target. In particular,
we want to change rose and pink colors to blue and gray, respectively. By simply projecting a correction color onto the target area,

the appearance does not match the target colors. However, by projecting a reddish environmental light over the other areas of thescene, observers perceive the intended colors.
4 C ONCLUSION AND FUTURE WORK
Projection mapping systems have a limited controllable color gamut
due to their additive feature. The goal of this work is to expand thiscolor gamut.
In this paper we describe the color gamut that can be controlled
by projectors and how colors that fall outside the color gamut of aprojection system can be presented by inducing color constancy. Ourcomputations assume that the color modification is a linear function.
In RGB space this only holds for the projector. A linear modificationof the color in RGB space does not necessarily translate to a linear
modification of the perceived color. In the future we aim to modify
our model to account for this. For example, the perceptual color
could be represented in the Lab color space, which is perceptuallylinear. Finally, we will conduct a user study to verify if our method
can expand the perceived color gamut of a mapping system.
We believe that our system can also assist people who sufferfrom color-vision deficiency. In the future, we want to investigate
if by manipulating the appearance with our system, color-deficient
observers can perceive colors they are not able to see naturally.
ACKNOWLEDGMENTS
This work was supported by ACT-I, JST.
R EFERENCES[1] R. Akiyama, G. Yamamoto, T. Amano, T. Taketomi, C. Sandor, and
H. Kato. Appearance Control in Dynamic Light Environments with aProjector-Camera System. In Proceedings of the IEEE VR Workshop onPerceptual and Cognitive Issues in AR, pages 1–6, 2016.
[2] T. Amano and H. Kato. Appearance Control Using Projection withModel Predictive Control. In Proceedings of the International Conference on Pattern Recognition, pages 2832–2835, 2010.
[3] O. Bimber, A. Emmerling, and T. Klemmer. Embedded Entertainment
with Smart Projectors. Computer, 38(1):48–55, 2005.
[4] Y. Fujimoto, R. T. Smith, T. Taketomi, G. Yamamoto, J. Miyazaki,
H. Kato, and B. H. Thomas. Geometrically-Correct Projection-BasedTexture Mapping onto a Deformable Object. IEEE Transactions onVisualization and Computer Graphics, 20(4):540–549, 2014.
[5] B. Jones, R. Sodhi, M. Murdock, R. Mehra, H. Benko, A. Wilson,
E. Ofek, B. MacIntyre, N. Raghuvanshi, and L. Shapira. RoomAlive:
Magical Experiences Enabled by Scalable, Adaptive Projector-Camera
Units. In Proceedings of the ACM Symposium on User Interface Software
and Technology, pages 637–644, 2014.
[6] T. Kawabe, T. Fukiage, M. Sawayama, and S. Nishida. Deformation
Lamps: A Projection Technique to Make Static Objects Perceptually
Dynamic. ACM Transactions on Applied Perception, 13(2):10:1–10:17,
2016.
[7] A. Madi and D. Ziou. Color Constancy for Visual Compensation ofProjector Displayed Image. Displays, 35(1):6–17, 2014.