IllumiRoom: Peripheral Projected Illusions forInteractive Experiences
Brett R. Jones1,2

Hrvoje Benko1

Eyal Ofek1

Andrew D. Wilson1

1

2
Microsoft Research
University of Illinois at Urbana-Champaign
Redmond, WA
Urbana, IL{benko, eyalofek, awilson}@microsoft.com
brjones2@illinois.edu

Figure 1. IllumiRoom is a proof-of-concept system that augments the physical environment surrounding a television to enhance interactive experiences. We explore the design space of projected visualizations which can negate, include or augment the surrounding
physical environment. (a) With a 3D scan of the physical environment we can (b) directly extend the FOV of the game, (c) selectively
render scene elements, (d) augment the appearance of the physical environment (here as a cartoon). All of the images in this paper
are un-edited; showing the real-time, working prototype.
ABSTRACTINTRODUCTION

IllumiRoom is a proof-of-concept system that augments thearea surrounding a television with projected visualizations
to enhance traditional gaming experiences. We investigate
how projected visualizations in the periphery can negate,include, or augment the existing physical environment andcomplement the content displayed on the television screen.
Peripheral projected illusions can change the appearance ofthe room, induce apparent motion, extend the field of view,and enable entirely new physical gaming experiences. Our
system is entirely self-calibrating and is designed to work in
any room. We present a detailed exploration of the design
space of peripheral projected illusions and we demonstrate
ways to trigger and drive such illusions from gaming content. We also contribute specific feedback from two groupsof target users (10 gamers and 15 game designers); providing insights for enhancing game experiences through peripheral projected illusions.

The television remains the focal point of living room entertainment today. While visual and audio quality has improved over the years, the content we watch (e.g., games,
movies, TV shows) remains boxed in the frame of the display and thus restricted by the physical size of the screen.
Also, the experience is limited to an entirely virtual world
trapped inside the television, ignoring the user’s surrounding physical environment.

Author Keywords

Spatial augmented reality; projection mapping; gaming;
immersion; apparent motion; augmented reality.
ACM Classification Keywords

H.5.1. Multimedia Information Systems: Artificial, augmented, and virtual realities

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee.
CHI 2013, April 27–May 2, 2013, Paris, France.
Copyright © 2013 ACM 978-1-4503-1899-0/13/04...$15.00.

In this paper, we propose a novel approach to enhance theviewing experience and blur the boundary between the onscreen content and the surrounding room. We extend thevisual experience outside of the television screen, using a
projector that covers a wide area of the surrounding physical environment. Similar to Focus+Context displays [3], thetelevision provides a traditional, high-resolution gaming
experience and the projector provides low-resolution information for the user’s peripheral vision. In contrast to previous work in Focus+Context (F+C) displays, we do not assume that the space surrounding the high-resolution display
is a flat, white projection screen. Instead, we capture theappearance and geometry of the surrounding room (e.g.,furniture, carpet, wallpaper) and use that information tocreate novel, interactive visual experiences.
In particular, our work investigates how projected visualizations in the periphery can negate, include or augment thephysical environment, and thus enhance the content displayed on the television screen. We call such visualizations
peripheral projected illusions. To investigate these illusions,
we implemented a proof-of-concept system, called IllumiRoom, which is capable of augmenting the environment
around traditional game content. IllumiRoom can change theappearance of the room, induce apparent motion, extend thefield of view (FOV), and enable entirely new virtual-physical gaming experiences. Our system is self-calibrating
and designed to work in any room. The concept could be
developed into a next generation game console with gamecontent designed from the ground up, or it could be an ‘addon’ for existing consoles with gamers writing scripts to‘mod’ their favorite games.
While our explorations are performed in the context of interactive games, the same or similar illusions could be used
to enhance movies and television content.
This paper makes the following contributions:• Implementation and description of the proof-of-concept
IllumiRoom system for real-time generation of peripheral projected illusions• Implementation of 11 illusions that demonstrate theimportant dimensions of the design space• Analysis of the design space of peripheral projected illusions• Discussion and implementation of methods to drive illusions from new and existing content• Feedback from 10 casual gamers and 15 expert gamedesigners over two separate evaluations; showing themost promising dimensions of the design space
MOTIVATING SCENARIO

To illustrate the idea behind the IllumiRoom system, imagine sitting down in your living room to play a video gameon your television. When the game starts, the room magically transforms to look like a cartoon, matching the shading in
the video game. The colors of the room become supersaturated and cartoon edges appear on your bookshelves.
You come across an enemy in the game, and suddenly astreaking bullet flies towards your character and then out of
the television. The enemy throws a grenade towards you.
The grenade rolls out of the television, bounces off yourcoffee table and explodes in your living room, throwing
shrapnel across your bookshelf. The entire living room appears to shake as you take damage from the explosion. Thisis just one of many interaction scenarios made possible withthe peripheral projected illusions described in this paper.
RELATED WORK

Our work draws on previous research in Spatial Augmented
Reality, artistic installations with ‘projection mapping’, and
HCI research on peripheral displays.
Spatial Augmented Reality

Spatial Augmented Reality (SAR) is the concept of merging
the physical and virtual worlds through video projection [8].
Shader Lamps first developed the idea of augmenting thecolors of physical objects with video projection; appropriating everyday physical objects as displays [24]. A number ofprojects have since demonstrated the use of projectors toaugment workspace environments [20,23,30].
Several SAR projects have enabled gaming experiences
involving physical objects, such as wooden blocks, which
can be arranged by users to create a game [15,30]. Two re-cent examples [4,30] supported real-time physics-based
interactions between virtual objects and physical objects.
We also draw on previous work in radiometric compensation, a process that compensates for the original color andgeometry of display surfaces [24]. A number of projects
have demonstrated turning a curtain or a building facade
into an almost ideal projection surface (e.g. [5,6]). Additionally, virtual images can be overlaid onto physical paintings [5,11] and the contrast of a printed image can be increased by superimposing projected content [7].
Most similar to our work, ‘Cartoon Dioramas in Motion’[25] demonstrated effects projected onto a small, white toy
car diorama. Effects included non-photorealistic shading,apparent motion through camera movement and virtual
lighting on the car. Inspired by this approach, we explore alarger design space including effects designed to work witha complex, textured living room. We also consider effects
that complement the content on the television.
Projection Mapping

Our illusions are also inspired by projection-based art installations, known within the art community as ‘projection
mapping’ or ‘video mapping’ [18,21]. These installations
are frequently used to make building facades come to life,
as in ‘Son et lumière’ shows. Artists create amazing effects
by rendering a video that is specific to the scene and projector location. In contrast, IllumiRoom generates real-time,interactive illusions that automatically adapt to any room,
any television and any game.
Display Research

The characteristics of a display can have a large impact onuser performance, especially while gaming. A larger display
has a wider field of view, can cause the user to feel more
immersed and more present in the experience, and is more
likely to induce simulator sickness [14,28].
Unfortunately, large displays can be expensive. To balance
the costs of a high-resolution, larger display with the benefits of a wider field of view, Baudisch et al. proposed Focus+Context displays. Focus+Context displays have a highresolution computer monitor (focus) surrounded by a lower
resolution projection screen (context). This configuration
takes advantage of the lower optical resolution in peripheral
vision [13]. We refer the reader to [9] for a more comprehensive review of Focus+Context techniques.
The Philips Ambilight television is an example of a commercially successful Focus+Context display. The Ambilight
TV has a series of color LED strips surrounding its edges.
The color of the LEDs is dynamically changed to match the
television content. In a follow-up research project, Philips
created a prototype with multiple LED strips located aroundthe room that change colors to adapt to gaming or movies[29]. Our system advances this idea to high-resolution projected content which is dynamically adapted to the appearance and geometry of the room.

A few previous projects have explored automatically extending video beyond the television frame for Focus+Context displays. The Infinity-By-Nine project [19]displayed extrapolated video content on three projector
screens surrounding a television, in a small CAVE [10] setup. Aides et al. described a system that creates an ultra-wide
foveated video from a normal input video sequence [1].
As in these previous works, the IllumiRoom system can beused for panoramic movies and television, and could be
extended to use extrapolated content from standard video
sources. Rather than relying on separate projection screens,
IllumiRoom adapts the projections to the user’s existing living room, and demonstrates a large range of possible peripheral projected illusions.
DISPLAYING IN THE PERIPHERY

The configuration of a small, high-resolution display withina large, surrounding projection is consistent with the setup
of human visual system. Compared to the center of gaze,
peripheral vision is highly sensitive to motion, but is relatively poor at distinguishing color and shape [27]. This isdue to the lower density of receptors and the changing distribution of receptor types outside the fovea. Optical flow inperipheral vision is thought to play an important role in theperception of motion [12]. Peripheral vision also enables theearly detection of objects of particular relevance (e.g. predators) before they come into full view.
In IllumiRoom, the smaller display serves the highresolution capability of the fovea, while the projector serves
the low-resolution, motion sensitive periphery. As peripheral vision is insensitive to color and highly sensitive to motion, many of the illusions are tailored to peripheral vision.
For instance, abstract patterns can be used to induce opticalflow and give an impression of motion.
The stimulation of peripheral vision could alternatively be
addressed by a single, very large display, a F+C screen [3]or a CAVE setup [10]. While these are great approaches fortraining and simulation, they are not suited for deployment
in typical living rooms. IllumiRoom does not require installing expensive projection screens, nor changing anything
about the existing living room. Furthermore, IllumiRoom
leverages the existing physical environment to create unique
virtual-physical game experiences.
ILLUMIROOM SYSTEM

Our vision for a fully developed IllumiRoom system includes an ultra-wide field of view device sitting on the user’s coffee table, projecting over a large area surrounding
the television (see Figure 2a). The device would be connected wirelessly to a next generation gaming console as asecondary display. The ultra-wide field of view could beobtained with an ultra-short throw projector, or by couplinga standard projector with a spherical/parabolic mirror (asthe peripheral content does not need a uniform pixel aspectratio). The room geometry could be acquired with a depth
sensor or a structured light scan (which would require only
a web camera, similar to the setups in [6,15]).

Figure 2. (a) Our vision for a productized IllumiRoom system
includes an ultra-wide field of view device that sits on a coffee
table and projects content in the area surrounding the television. (b) Our current proof-of-concept prototype uses an offthe-shelf projector and a Kinect sensor.

Our current proof-of-concept prototype uses a commodity
wide field of view projector (InFocus IN126ST) and a Microsoft Kinect for Windows sensor (see Figure 2b). The
prototype is limited by the field of view of the Kinect andprojector (~57º horizontal FOV). Therefore, the system ismounted above and behind the user’s head, as they are seated on a couch in front of the television.
The Kinect sensor captures the color and geometry of thescene, and the system renders the illusions using the acquired depth map. Careful calibration of the system is required in order for the illusions to tightly match the onscreen content and the physical environment. The calibration of the IllumiRoom system is fully automatic, determining the relative pose of the projector to the depth sensor andthe position of the television in the projector image. Therefore, setup only requires that the projector and depth camera
are placed such that they cover the area surrounding the TV.
The automatic calibration projects Gray code sequences [2]to establish dense correspondences between the projector
and the Kinect’s color camera. Using the default parameters
for the Kinect, we transform these correspondences into 3D
points. We then solve for the intrinsic and extrinsic parameters of the projector. As there is no depth data for specularsurfaces such as that of shiny, black televisions, we recover
the position of the TV using a 2D homography, with virtual
markers displayed in each corner of the screen. It is important to note that the calibration need only happen once.
Thus the Kinect camera could be used to calibrate the projector and then returned to the TV stand for gaming.
The acquired living room geometry is used during calibration and to drive illusions that respond to living room furniture in a physically realistic manner (similar to the ideas in[4,30]). IllumiRoom could easily be extended to continuously capture the geometry, enabling ad hoc changes to theliving room such as moving furniture and people.
Radiometric Compensation

IllumiRoom projects on the existing living room furniture,and therefore cannot rely on the presence of a white, flatprojection surface. Using radiometric compensation [5,7,24]
the projected light may be modulated to achieve a desiredcolor, given knowledge of the existing surface color andgeometry. However, this process is limited by the brightness, dynamic range and color primaries of the projector, asFigure 3. Projecting wider field of view game content around the TV can increase immersion. (a) F+C Full: The full game content is
projected with radiometric compensation onto the surrounding furniture. (b) F+C Edges: A high-contrast version where only theedges in the game content are projected. (c) F+C Segmented: Game content is projected only onto the flat, rear wall. (d) F+C Selective: Selective game content (e.g. bullets) bleed out of the TV.

well as with the existing surface color, material, and geometry. Accordingly, many desired surface colors are unachievable. For example, a solid red object in our physicalenvironment cannot be made to appear green or blue.
Therefore, all of the illusions use the fact that they occur inperipheral vision. Illusions that may not seem realistic onclose examination may be quite effective in the user’s peripheral vision. Most illusions do not attempt to accurately
reproduce a scene, but aim to introduce motion in the periphery. Also, subtly modifying the existing surface color
may be more effective than replacing the color entirely.
Illusions that modify the surface color and not the apparent
geometry of the room are independent of viewing position,and thus can be viewed by multiple users from any position
within the room. Illusions that add to or modify the geometry of the room are inherently dependent on viewing position and are most effective when viewed from a fixed point
within the room. Generally, as the room’s furniture becomes
more complex, effects which virtually modify the geometry
are more sensitive to viewing position.
ILLUMIROOM ILLUSIONSWe envision the IllumiRoom system supporting a great variety of peripheral projected illusions, limited only by theimagination of game designers. We have prototyped illusions that we believe represent the primary dimensions of
the design space (discussed in the next section). This represents an initial survey of the design space and is not exhaustive. We outline the techniques used to create the illusions,and describe the illusions that were implemented. The illusions are best understood by demonstration (see the accompanying video).
We present the illusions as distinct points in the design
space. However, the illusions can be combined in the IllumiRoom system. While some illusions are mutually exclusive (F+C Full & F+C Edges), almost all illusions can be
combined.
Focus+Context

The most obvious way to increase the user’s sense of immersion is to extend the content from the television screen

out into the room, replacing the physical reality with thegame’s reality. We call this Focus+Context Full (F+C Full)(see Figure 3a). This is essentially the Focus+Context approach [3], but using a non-flat, non-white projection surface, with radiometric compensation. As stated above, theability to compensate for the existing surface color is limited, and so this effect relies on the fact that the user will
directly attend to the television screen rather than the periphery.
The next illusion, Focus+Context Edges (F+C Edges), provides a maximum contrast representation of the peripheral
content by displaying black-and-white edge information(Figure 3b). This illusion is more robust to ambient light in
the room, and increases peripheral optical flow.
Focus+Context Segmented (F+C Seg) extends the gameonly onto the rear wall surrounding the television (Figure3c). While the furniture in the room may not be ideal for
projection, most televisions are at least partially surrounded
by a flat, light colored wall which provides an ideal, viewindependent projection surface. This rear wall could befound by a recursive RANSAC plane fitting procedure [22],seeded with the location of the television. Our prototype
currently relies on a manually-specified image mask identifying the rear wall.
While the previous illusions bleed all game information into
the physical environment, there are times when a more subtle approach may be appropriate. For example, when watching a horror movie, an atmosphere of immersion and suspense is created by the absence of the villain, and then theirsudden appearance. When all information is present all thetime, there is no opportunity for suspense or surprise. With
Focus+Context Selective (F+C Sel) only certain game elements escape the television (Figure 3d). For instance, afirst-person shooter might bleed only weapons fire or explosions out of the television. Or, markers representing othercharacters or key items in the game may be selectively displayed, thus increasing their emphasis.
These illusions increase immersion, induce apparent motion,and provide additional information about the game content.

The Focus+Context illusions all require access to the video
game’s rendering process.

noir” look. We leave more complex texture replacement
techniques (i.e., procedural texturing) for future work.

Peripheral Flow

Texture Displacement

If we do not have access to the rendering process, apparent
motion may still be induced through peripheral visual flow,
as long as we have information about the motion of thegame’s camera. In Grid (Figure 4a), an infinite grid moves
with the motion in the game. When the user moves or looks
left or right, the grid moves in the opposite direction. Whenthe user moves forwards or backwards, the grid zooms in orout.

Imagine an explosion in a video game or film that is so
powerful that it shakes your entire room. IllumiRoom canuse projected light to modify the appearance of objects in
the room, making them appear to move or change size, or
distort in more extreme ways. Due to the projector’s limited
ability to compensate for surface color, these illusions aremost effective if the displacements are small and shortlived. Radial Wobble (Figure 6) distorts the apparent room
geometry in a ‘force field’ ripple effect. The illusion displaces the original surface texture in an expanding, radialsinusoidal wave.

Figure 4. Apparent motion may be induced without access togame rendering by displaying abstract peripheral visual flow.
(a) An infinite Grid moves with the game’s camera. (b) A 3D
Starfield of spheres moves with the game’s camera.

Similarly, Starfield (Figure 4b) creates peripheral flow bymoving 3D points to match the movement of the game’s
camera. Points are randomly generated in the virtual space
between the TV and the user, according to the position andorientation of the game’s camera.
Color Augmentation

The physical environment may be augmented to match thetheme or mood of the game. For instance, room colors may
be super-saturated to resemble a cartoon look by simply
projecting the color of the surface back onto itself [7], andthen drawing silhouette edges black (Figure 5). Similarlycolors may be desaturated to create a black-and-white, “filmFigure 5. Appearance: The appearance of the physical environment may be augmented (a), to match the theme of thegame. (b) Cartoon super-saturated colors and silhouette edges(c) Black-and-white (de-saturated) colors.

Figure 6. Radial Wobble: To create a ‘force field’ ripple effect,
the physical surface texture is displaced with a radial expanding sinusoidal pattern.
Lighting

With full control over the illumination, the lighting in thephysical environment may be altered [24,25]. Room lighting can change based on the mood or theme in the game.
For example, the look of a scene set in space might beachieved by illuminating with point light sources and harshshadows. The Lighting effect attempts to match the lighting
conditions of the physical environment with the virtual environment (Figure 7). Our prototype implements soft shadows and a single point-light source, matching the closest
point-light source in the game.

Figure 7. Lighting: The lighting and shadows of the physicalenvironment are matched to lighting in the virtual environment.

Physical Interactions

In Focus+Context Selective elements from the game (e.g.,bullets) break the screen barrier and fly into the physicalenvironment; however, there are no physical interactions
with the environment. Bounce envisions an element in thegame leaving the virtual environment and interacting withphysical reality. In our prototype, a grenade rolls out of the
television, and then bounces and rolls around the physicalenvironment according to a physics simulation using theroom geometry (Figure 8).

n

sio

er

m

Im

Pr

e
es

nc

n
n
n
tio
io
tio
nt
nt ma lity ec lity tortere ion
e
r
n
t
a
s
t
n fo ea n ea ip oem
Co In R Co R D
Ap M
Th

e

F+C Full
F+C Edges
F+C Segmented
F+C Selective
Grid
Starfield
Appearance
Radial Wobble
Lighting
BounceSnow

Figure 9. IllumiRoom illusions have a variety of goals: increasing immersion and presence, inducing apparent motion,
providing additional information or content, creating a senseof atmosphere or theme and supporting entirely new physicalvirtual game mechanics.

Figure 8. Virtual objects can realistically interact with physicalobjects. (a) In Bounce a grenade rolls out of the TV andbounces around the living room. (b) In Snow, falling snow
moves according to the user’s movement in the game and interacts with the physical environment.

In Snow, falling snow interacts with the physical environment, briefly collecting on surfaces in the room. Similar toStarfield, the snow moves according to the movement of theuser in the virtual environment, allowing the user to walk or
drive through snow (Figure 8). Our prototype simulates theaccumulation of snow by turning surfaces with an upwardfacing normal gradually white.
DESIGN SPACEThe range of possible peripheral projected illusions is as
great as the range of visual effects in video games or film.
In order to better understand the full range of illusions, we
identify the primary dimensions of the design space. Illusions are defined by three primary factors: the goals of theillusion, the connection or separation from physical reality
and the level of abstraction from the game content.
Illusion Goals

Peripheral projected illusions can enhance gaming experiences in a variety of ways. All of the proposed illusions attempt to increase the user’s sense of immersion by increasing the FOV and surrounding the user with content. All of
the illusions can increase the user’s sense of presence, making the user feel ‘in the game.’ More specifically, the illusions can induce apparent motion, provide additional information and content, create a sense of atmosphere or themeand support entirely new physical-virtual game mechanics.
These goals are not mutually exclusive, and a single illusion
may support multiple goals (see Figure 9).
Connection to Reality

IllumiRoom enables designers to explore the entire spectrumof the reality-virtuality continuum [17]; by negating, includ-ing, or augmenting the surrounding physical environment.
Designers can negate the physical environment, making theliving room disappear; thereby immersing the user in an
entirely virtual environment (e.g., F+C Full, F+C Edges,
Grid, Starfield).
Or the game can be grounded in reality, by including thephysical environment and selectively introducing virtual
objects into physical reality (e.g., F+C Seg, F+C Sel,
Bounce, Snow).
Finally, IllumiRoom enables a unique middle ground, where
the physical surroundings are augmented to match the virtual environment, creating a mixed reality where it is not clear
what is real or virtual (e.g., Appearance, Lighting, Radial
Wobble).
Abstraction

The illusions can be dependent or independent of the underlying game content. For instance, if we extend the exact
game content outside of the TV, the illusion is entirely dependent on the game content (e.g., F+C Full, F+C Edges,
F+C Seg, F+C Sel, Bounce). However, if we display an
abstract visualization, such as a moving grid, then the peripheral illusion is independent of the game content (e.g.,
Grid, Starfield, Snow, Appearance, Lighting, and Radial
Wobble).
TRIGGERS

The illusions can be connected with game content through avariety of means. Ideally, IllumiRoom would be directly
integrated into a next generation console and new games
would be designed for IllumiRoom from the ground up. We
envision an API that enables triggering illusions, changing
surface appearance, controlling room lighting, inserting
objects into the physical environment, etc.
In our prototype, we also demonstrate how to connect existing game content to drive illusions. In order to make thesystem work with numerous games, the IllumiRoom system
is a separate component from the game examples, and networking messages are sent to share information. We present
a variety of methods for modifying existing content. Figure

Good
Excellent

Co
nt
Inp roller
utCont
Ru roller
mb
le
Op
tica
Fl o lw
Sour(Ne ce
two
rk M
Sou
sgs
rce
)
(Re
nde
ring
Un
)
ifie
(Ide dal)LegendF+C Full
F+C Edges
F+C Segmented
F+C Selective
Grid
Starfield
Appearance
Radial Wobble
Lighting
BounceSnow

Figure 10. Ideally the IllumiRoom system is unified with thegame content. However, existing game content can be usedwith the prototype through a variety of input sources. Here weshow the different ways of connecting game content to the illusions.

10 enumerates the input sources that can be used for eachillusion.
If the source code of the game is available, then the gamecan be easily modified to trigger effects. In the current prototype, the game sends networking messages to the IllumiRoom system in order to toggle illusions on and off.
The position and orientation of the player (or often, equivalently, graphics camera) in the virtual world may be sent todrive the Peripheral Flow illusions. With access to sourcecode the game can support Focus+Context illusions byconducting a second rendering pass using a wider field ofview. In our prototype this information is sent to the IllumiRoom system through a texture residing in sharedmemory.
Source-code is not available for most existing games. Fortunately, many of the illusions can be controlled by othermeans. For example, the Peripheral Flow illusions need
only some measurement of the player/camera movement.
An approximation can be obtained by real-time image analysis of the rendered game (e.g., optical flow). In our prototype, any game’s image may be captured using a video capture card (DeckLink Intensity Pro), and subsequently analyzed by optical flow techniques to drive the illusions.
A significant amount of game state may be inferred solely
from the user’s input through the controller. Givenknowledge of the game and the controller mapping, controller input may be mapped directly onto illusions. For instance, if the game is a first-person shooter and the playerpresses the ‘Right Trigger’ button on the XBox controller,
then the player must be shooting. The Radial Wobble illusion may then be activated, for example. In our prototype,
input from the Xbox controller and vibrotacticle output sent
to the controller from the game can drive illusions. This
works with any game. We envision users scripting and sharing ‘mods’ of their favorite games to work with the IllumiRoom system.

For our prototype, games are run on the same PC as the IllumiRoom system. GlovePIE 1 intercepts and sends gamecontroller input to the IllumiRoom system by networking
messages. Vibrotactile output at the controller is captured
by using an Xbox controller modified with an Arduino.
END USER EVALUATION

We evaluated eleven points in the design space of peripheral
projected illusions through a user study. We were primarily
interested in determining user perceptions along different
dimensions of the design space. Just as there is no ‘best visual effect’ for film or video games, we were not expecting
to identify a ‘best illusion’. Rather, we sought the strengths
and weaknesses of each technique. We were also interested
in the user’s overall comfort level during the illusions. Each
user interacted with the illusions, answered a Simulator
Sickness Questionnaire (SSQ) [16], and performed a cardsorting task where they rated and ranked each illusion along
a variety of dimensions.
We recruited 10 participants (ages 20-30, 2 female). In orderto ensure that participants could focus on the gaming experience and not the control scheme, we required that participants be familiar playing first-person shooter games with an
Xbox controller. The user study lasted approximately one
hour and participants received $10 for their participation.
Audio and video of the sessions were recorded for the purposes of extracting quotations.
In our experiments, the user sat on a couch 8 ft. from a 40”
diagonal television displaying 720p content. A projector
was mounted above and slightly behind the user’s head, 11ft. from the television and 5 ft. above the ground, at a resolution of 1280x800, casting a 130 inch diagonal image(measured within the plane of the television). The televisionviewing angle was 20º and the projector’s viewing angle
was 45º. These distances were chosen to maximize the projected area, within the limits of the working range of theKinect sensor.
Game Content

Each participant interacted with eleven illusions paired withmatching game content. We paired game content with theillusions in order to maximize the effectiveness of the experience. The majority of the illusions were paired with an
open-source first-person shooter (Red Eclipse2). This created a rich, interactive experience, enabled by access tosource code. The Snow illusion was paired with a racing
game with a snow level (SuperTuxKart 3 ), triggered with
controller input. Lighting was illustrated by a Unity3D race
car example4, which uses a few clear light sources. The light
positions were sent by network messages. The Bounce example was paired with a custom DirectX application in order to perfectly synchronize the content between the physical and virtual worlds. This example was not interactive, but
served to demonstrate the basic concept of virtual to physi1

http://glovepie.org/glovepie.php
www.redeclipse.net
3
http://supertuxkart.sourceforge.net
4
http://unity3d.com/support/resources/tutorials/car-tutorial
2

Overall Satisfaction
Unsatisfied

NeutralSatisfied

It was fun
Disagree Neutral

It felt sick
Agree

Disagree Neutral

I was moving
Agree

Disagree Neutral

Agree

I was in the game The game and the room
were connected
Disagree Neutral

Agree

Disagree Neutral

Agree

F+C Full
F+C Edges
F+C Segmented
F+C Selective
Grid
Starfield
Appearance
Radial Wobble
Lighting
BounceSnow

Figure 11. Mean rating for each illusion evaluated in the user study. Participants rated ‘overall satisfaction’ and responded to fivequestions on a five point Likert scale from ‘strongly disagree’ to ‘strongly agree’.

cal interactions for users. See the accompanying video forthe exact content.
Procedure

Participants interacted with the eleven illusions with paired
video game content in a randomized order. Participants werefirst introduced to each game without any peripheral projected illusions so that they could learn the control scheme
and game objectives. Then, the illusion was revealed andgameplay continued. Immediately following the last illusion, participants filled out a Simulator Sickness Questionnaire (SSQ) evaluating their overall comfort level.
Participants then engaged in a card sorting task, where they
rated the illusions on various dimensions. Cards with a picture representing each illusion were placed on a coffee tablein front of the user to act as a visual reminder of the illusion. The participants rated the ‘overall satisfaction’ of theillusion on a 5 point Likert scale, and then ranked its ‘overall satisfaction’ (without ties). This procedure was designed
to spur careful critique and comparisons, not to identify a‘best illusion.’ Participants then rated the illusions on a 5
point Likert scale in response to the following prompts: “It
was fun”, “I felt sick”, “I felt like I was moving”, “I felt like
I was in the game world”, and “The game and the physicalworld felt like separate spaces.” As this task was completed
after viewing all illusions, this enabled the users to mentallycompare and contrast the illusions. Users did not express
any concern regarding their ability to recollect the illusions.
Results & Feedback

Participants responded very positively to the illusions andthe concept of IllumiRoom. As expected, there was not one‘winning’ illusion, but a diverse set of illusions that havestrengths on different dimensions of the design space. For asummary of participant responses see Figure 11.
The results from the SSQ give a coarse indication of potential issues with simulator sickness. The mean value of Total
SSQ (14.96), was within normal ranges for playing stereoscopic video games [26] or interacting with immersive simulators [16]. To keep the user study a reasonable length we
did not compare SSQ scores across illusions. Future work
could explicitly compare individual SSQ scores.

Participants found the Focus+Context techniques some of
the most useful illusions for game play, because they naturally provide additional gaming information. Simply byextending the field of view of the game, these illusions increase the user’s sense of motion and presence (“I was in
the game”). For instance, one user commented that F+C
Selective was “very useful for finding items, it kinda lets
you see just pieces of the game that are important to know
around you, and do spatial orientation.” However, the F+C
illusions can also make the user overwhelmed or more
prone to sickness. For instance, one user commented thatwith F+C Full, “There was just a lot going on. To see everything going on around you it was like really cool but especially in as fast paced a game as that, it was just too muchhappening at once.” F+C Edges only made that problem
worse, by emphasizing high contrast edges; thereby receiving the lowest satisfaction scores. The F+C Selective and
F+C Segmented illusions seem to represent a middle
ground between immersion, adding content information andbalancing user comfort. As one user said about F+C Selective, “Out of all of these that would be the most useful andimmersive without being distracting.”
Of the Peripheral Flow illusions, Grid seemed to be more
successful at imparting a sense of apparent motion; possibly
due to the more coherent, 2D nature of the visual flow thatit created. One user commented about Starfield, “I like theidea of being able to feel your motion in the game, but the
random placement of points made it less consistent.” Users
were split as to whether the Peripheral Flow illusions weremore or less distracting than the F+C illusions. Multiple
users commented that they had a sense of motion, but
missed the contextual information of the F+C illusions.
Appearance worked well at creating a sense of atmosphere
for the game. As one user put it “The appearance is awesome. Like that could just be an effect in someone’s livingroom all the time. That is just super cool.” However, theeffect by itself was deemed not as interesting as it would be
if it were combined with other effects. One user stated “Ifyou are going to project something, you might as well project something that is directly relevant to the game.”Radial Wobble was another ‘magical’ effect, that “kinda
made you feel like you are physically doing something in[the game].” However, some users said the effect was offputting, because “all your stuff is shaking in the real world.”
In the evaluation, Radial Wobble was triggered in the firstperson shooter every time a weapon was fired. As one userstated, “It just seemed like one that you should save for really impactful moments. So I just wouldn’t want it all thetime.”
Participants generally liked the play between the physicaland virtual worlds in the Bounce, Snow and Lighting illusions. For example, Lighting served to create a better senseof immersion and presence for some users “I think more sothan any of the others, that one made me feel like I was in
the game world.” For the Bounce illusion, due to the roughnature of the paired DirectX sample application users had to
imagine what it would be like to incorporate this effect intoa real game. Multiple users referenced the grenade indicatorin Infinity Ward’s Call of Duty, “Most of the time I don’t
see the grenades at all...They always kill me, so that was agood one.” The Snow illusion created a sense of atmosphereand movement for users; however most users did not notice
the physical interactions with the geometry. This is because
the snow quickly ‘melted’ after impact. One user commented that they “really wanted it to pile up more. I found it to
be melting and if it was piling up it would be an indicationof how long I was playing and that would be cool.” However, these illusions (particularly Lighting) easily revealed
even the smallest mismatch between the game world andthe real world, which hampered the illusions for the participants. One user commented that a better use for Lighting
would be in illustrating the difference in illumination when
the player moves between indoors and outdoors in an open
world game (e.g., Grand Theft Auto).
GAME DESIGNER FEEDBACK

In addition to gamers, we also elicited informal feedback
about our illusions from game designers (N=15). From theprevious user study, we grasped how gamers would react tothe illusions, but we wanted to see what content creators
thought about the design space. We showed the illusions tothree groups, each composed of five game designers orgame artists. Each participant had more than 5 years of experience in the game industry.
While individual game designers had different preferences,
we extracted the commonalities amongst designers. Generally, the game designers thought that F+C Full was impressive at first, but “would probably not hold up to the longterm play.” They thought the most promising illusions wereones that altered the mood (Appearance), altered the players
sense of reality (Radial Wobble), or ones that selectively
showed game content without becoming overwhelming(F+C Selective, F+C Segmented, Bounce). They expressed
concern over illusions that need a tightly, fine-tuned connection with the game (Lighting); as they were worried
about destroying the illusion for the user if the illusion was
not perfectly matched to the game content. Therefore, they
saw the merits of Snow and Starfield, which have much

Figure 12. IllumiRoom can also be used for panoramic video.
(a) A custom dual camera rig for capturing F+C Video (a). The
panoramic video content can be displayed as (b) F+C Full, (c)F+C Segmented. (d) Peripheral content can be deemphasizedby blurring (shown with F+C Full).

looser connections with the source content. They all suggested games where IllumiRoom could be applied (e.g.,
Forza, Skyrim, Halo, Portal) and enthusiastically suggested
that they would like to have a setup like this at home.
BEYOND GAMING

All our examples so far have focused on computer games;
however, IllumiRoom can also be used to enhance the viewing experience of film and television. In such applications,the user is not in direct interactive control of the experience,but the dual nature of the content can create some interesting cinematic effects. For example, imagine watching asporting event (e.g., football game) where the ‘focus’ video
is tightly held on the players capturing their actions, whilethe ‘surround’ video is immersing the viewer with live panoramic views of the stands and the fans at the game. Or imagine a music video where the focus and surround video
show different footage, but mimic the camera motion (e.g.,
pan with exactly the same speed and direction).
To begin experimenting with such video experiences, webuilt a custom dual camera rig, with a narrow FOV ‘focus’
camera and a wide FOV ‘surround’ camera (Figure 12).
This panoramic content can be displayed around the television using F+C Full or F+C Segmented (with radiometriccompensation). Additionally, it can be blurred to decrease
the emphasis on the peripheral content. Future video content
could be filmed with such a rig, or existing video content
could be extrapolated (e.g. [1,19]). Our existing experience
with recording and watching such dual feed videos in IllumiRoom leads us to believe that a whole new set of cinematic effects could be produced by creatively combining thetwo video feeds.
CONCLUSION AND FUTURE WORKIn this paper we presented a proof-of-concept system foraugmenting the physical environment surrounding a television to create a more magical gaming experience. We presented eleven peripheral projected illusions that demonstrate the variety of the design space. We then elicited feedback from gamers and game designers about the peripheral
projected illusions along dimensions of the design space.
While we did not formally compare the experience of theIllumiRoom system with a standard TV setup, the very positive feedback of both user groups indicates that there isgreat promise in the approach.
While we explored eleven points in the design space, there
are many peripheral projected illusions left to be explored.
We only scratched the surface of what is possible with ashared physics representation between the virtual and physical environments in Bounce. It is not clear what it means if
a grenade falls out of a game and explodes in your living
room; particularly, if the virtual camera is moving aroundand the physical environment is static. Furthermore, assome illusions distort reality, future work should explore
how far users are willing to push their concept of reality, intheir own living room.
Another promising direction is to further explore illusions
specifically for video content (e.g. TV shows or full lengthfeature films). Authoring such content is a challenge, but it
could enable unique viewing experiences. Can a grenadefrom the latest Bond film explode in your living room?
How would such content be authored? It would be important to investigate how the movie director should deal
with the fixed nature of a film and the randomness imbued
by the system adapting to the user’s living room.
Additionally, our methods for connecting content and illusions are not exhaustive. It should be possible to use othercues (e.g., audio) to trigger illusions. For instance, a loud
explosion in a film could trigger a Radial Wobble illusion.
Simple techniques such as these could enable users to ‘mod’
existing films and games, and share the scripts. Future work
could explore this design process from a standpoint of collaborative design or crowd-sourcing.
Finally, before the IllumiRoom system can be in every livingroom, the final form factor, cost and computational requirements of the system must be determined. While there
are many unanswered questions regarding peripheral projected illusions, we hope we have demonstrated that they
are worth answering.
REFERENCES
1. Aides, A., Avraham, T., and Schechner, Y.Y. Multiscale
Ultrawide Foveated Video Extrapolation. IEEE ICCP, (2011).
2. Batlle, J., Mouaddib, E., and Salvi, J. Recent progress incoded structured light as a technique to solve thecorrespondence problem: a survey. Pattern recognition 31, 7(1998), 963–982.
3. Baudisch, P., Good, N., and Stewart, P. Focus Plus Context
Screens: Combining Display Technology with Visualization
Techniques. ACM UIST, (2001).
4. Benko, H., Jota, R., and Wilson, A. MirageTable : FreehandInteraction on a Projected Augmented Reality Tabletop. ACM
CHI, (2012).
5. Bimber, O., Coriand, F., Kleppe, A., Bruns, E., Zollmann, S.,
and Langlotz, T. Superimposing pictorial artwork withprojected imagery. IEEE Multimedia 12, 1 (2005), 16–26.
6. Bimber, O., Emmerling, A., and Klemmer, T. Embedded
entertainment with smart projectors. IEEE Computer 38, 1(2005), 16–26.
7. Bimber, O. and Iwai, D. Superimposing dynamic range. ACM
Trans. on Graphics 27, 5 (2008).

8.
9.
10.
11.
12.
13.
14.

15.
16.

17.

18.
19.
20.
21.
22.
23.

24.

25.
26.
27.
28.
29.
30.

Bimber, O. and Raskar, R. Spatial augmented reality:
Merging real and virtual worlds. AK Peters Ltd, 2005.
Cockburn, A., Karlson, A., and Bederson, B.B. A review ofoverview+detail, zooming, and focus+context interfaces.
ACM Computing Surveys 41, 1 (2008), 1–31.
Cruz-Neira, C., Sandin, D.J., and Defanti, T.A. SurroundScreen Projection-Based Virtual Reality : The Design and
Implementation of the CAVE. SIGGRAPH, (1993).
Flagg, M. and Rehg, J.M. Projector-Guided Painting. ACM
UIST, (2006).
Gibson, J.J. The Perception of the Visual World. Boston:
Houghton Mifflin, 1950.
Henderson, J. Human gaze control during real-world scene
perception. Trends in Cognitive Sciences 7, 11 (2003).
Hou, J., Nam, Y., Peng, W., and Lee, K.M. Effects of screen
size, viewing angle, and players’ immersion tendencies ongame experience. Computers in Human Behavior 28, 2(2012), 617–623.
Jones, B., Sodhi, R., Campbell, R., Garnett, G., and Bailey,
B.P. Build Your World and Play In It: Interacting with
Surface Particles on Complex Objects. IEEE ISMAR, (2010).
Kennedy, R.S., Lane, N.E., Kevin, S., and Lilienthal, M.G.
Simulator Sickness Questionnaire : An Enhanced Method for
Quantifying Simulator Sickness. Aviation Psychology 3, 3
(1993), 203–220.
Milgram, P., Takemura, H., Utsumi, A., and Kishino, F.
Augmented Reality: A class of displays on the realityvirtuality continuum. Telemanipulator and Telepresence
Technologies 2351, (1994).
Mine, M., Rose, D., Yang, B., Jeroen van Baar, andGrundhöfer, A. Projection-Based Augmented Reality in
Disney Theme Parks. IEEE Computer 45, 7 (2012), 32–40.
Novy, D. and Jr., V.M.B. Infinity-by-nine.
http://obm.media.mit.edu/, (2012).
Pinhanez, C. The Everywhere Displays Projector: A Device
to Create Ubiquitous Graphical Interfaces. ACM UbiComp,(2001).
Projection Mapping Central. www.projection-mapping.org. .
Quirk, P., Johnson, T., Skarbez, R., Towles, H., Gyarfas, F.,
and Fuchs, H. RANSAC-Assisted Display Model
Reconstruction for Projective Display. IEEE VR, (2006).
Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., andFuchs, H. The Office of the Future: A Unified Approach to
Image-Based Modeling and Spatially Immersive Displays.
ACM SIGGRAPH, (1998).
Raskar, R., Welch, G., Low, K.L.K., and Bandyopadhyay, D.
Shader lamps: Animating real objects with image-based
illumination. Eurographics Workshop on Rendering
Techniques, (2001), 89.
Raskar, R., Ziegler, R., and Willwacher, T. Cartoon dioramas
in motion. ACM NPAR, (2002), 7.
Schild, J., Laviola, J.J., and Masuch, M. Understanding User
Experience in Stereoscopic 3D Games. ACM CHI, (2012).
Sekular, R. and Blake, R. Perception, 2nd Edition. 1990.
Tan, D.S., Gergle, D., Scupelli, P., and Pausch, R. Withsimilar visual angles, larger displays improve spatial
performance. ACM CHI, (2003).
Weffers-Albu, a., De Waele, S., Hoogenstraaten, W., andKwisthout, C. Immersive TV Viewing with Advanced
Ambilight. IEEE ICCE, (2011), 753–754.
Wilson, A., Benko, H., Izadi, S., and Hilliges, O. Steerable
Augmented Reality with the Beamatron. ACM UIST, (2012).