26
90

特集

日本バーチャルリアリティ学会誌第 19 巻 2 号 2014 年 6 月

特集 ■ プロジェクションマッピング

プロジェクションマッピング

Projection Mapping

Challenges of Projection-Based Augmented
Reality Systems in Theme Park Attractions

Anselm Grundhöfer（Disney Research Zürich）

1．Introduction

to enable sufficiently accurate geometric and photometric

The last decade has witnessed tremendous improvements incalibration for content generation of such installations, thedisplay technology. Today’s smartphones and laptops offer

question still remains as to whether non-experts will be able

screen resolutions far beyond what the human eye is able

to successfully apply and judge calibration, as well as know

to resolve. TV systems are also in the process of changing

how to maintain such a complex system without specificto resolutions far beyond “full HD”, and cinemas haveknowledge as to its underlying algorithms.

almost completely transitioned from analog film to digitalprojection. The high-quality projectors of today’s cinema

2．A Quick Introduction to Projection Basedare also well suited for non-standard setups such as, for

Augmented Reality

example, in theme park environments. Generating a scenic

While already an active research field for several decades,

lighting environment is one of the most important factors

augmented reality（AR）has gained widespread attention overin the overall immersion quality of such attractions, and thethe past few years. For most people today, the term “augmented

usage of digital projectors can help to raise the bar in thisreality” refers to video see-through AR applications thatregard. It is to be expected that more and more projection

include perspectively corrected and location-aware graphics

systems will be used in the future not only in theme parks,

or information. Alternatively, projection-based AR belongs to

but also for general entertainment, advertisements, and art

the group of spatial AR technologies in which augmentation is

installations. While the tools have already been developed

directly visible in space instead on a screen. While this requires
more complex system setups compared to, for example, a cell
phone with an integrated camera, it offers several interesting
applications, such as for building projections, scenic lighting
effects, and the superposition of high-quality textures toenhance or alter the appearance of real world surfaces.
Prominent examples are given in [2], [3] and [6].
When projecting imagery accurately onto complex-shaped
objects, several problems have to be overcome to achieve a
convincing impression. Solutions for how to solve such issues
will be quickly summarized below (interested readers should

Figure 1
The “Disney Dreams” show at Disneyland Paris uses projectors to turnthe whole “Castle of The Sleeping Beauty” and its surrounding into agigantic show screen.

see Bimber et. al.’s report [1] for a more detailed overview ofthe fundamental algorithms).

26

特集

プロジェクションマッピング

JVRSJ Vol.19 No.2 June, 201427
91

projected image edges. Given that light contribution
adds up in overlapping areas, though may abruptly dropin the case of shadow casts, the individual contributionsof projectors have to be adjusted to equalize reflected
intensities with non-overlapping areas. This requires thegeneration of smooth transitions to avoid the appearance of
hard intensity edges in the case of slight misregistrations.
Solutions for this problem are presented in [2]. To correct
for the non-linear response functions of projectors, which
are usually set to sRGB or a similar gamma curve, aFigure 2“The Legend of Captain Jack Sparrow” at Hollywood Studios in Orlando
uses projectors to immerse patrons in 360-degree projections on screens,rocks, and a pirate ship.

radiometric measurement has to be carried out beforehand
to linearize its response function.
・Photometric Calibration and Compensation:

・Geometrical Registration: Projecting onto complex

Equalizing the color gamut of multiple projectors [ 5 ] asshapes requires the accurate mapping of each individual

well as neutralizing a surface’s varying spatial reflectanceprojector pixel onto a given surface. Most real-world surfaces[ 7 ] requires careful photometric device calibration andare far from perfectly flat planes. Imagine, for example, acompensation image calculation. In addition to themodern car chassis or natural surface. To accurately project

mentioned issues, several specialized methods exist toaugmented content onto such surfaces, a precise model ofalso compensate for more complex effects such as global

the surface geometry has to be attained. This can either

illumination, specular reflections, and textures. Some of

be carried out from existing CAD data or other sources,

these approaches deal with individual problems, while

such as laser scanners, or can be directly generated by theothers try to solve for all unwanted effects at once [8].

projection system by adding cameras to it. Such setups are

Practically it is sufficient to analyze a system to figure out

usually referred to as projector-camera-systems or procams-

which illumination effects have the greatest bearing on

systems. In these, structured light patterns are projected andimage quality, and then trying to compensate for each of

captured by the cameras to generate a quick and accuratethese individually by using specialized methods adapted formapping between individual camera and projector pixels.

the source of image degradation. Regardless of the setup,

The most widely used patterns are gray codes or other types

diffuse scattering commonly occurs close to surface edges,of spatially- or color-coded patterns. Depending on the

yet this can be compensated for in various ways as well.

application, the mapping of projector pixels onto a surface

・Synchronization: Synchronization is a technical issue

can then be carried out by using 2D lookup tables between

that has to be solved to generate a compelling projection.

camera and projector image planes, or by applying fullCameras must be synchronized to projectors in ordergeometric calibration and treating each individual projector

to capture the correct structured light patterns during

as a pinhole device equivalent to a camera (correcting forcalibration. This problem can be solved easily by simply

lens distortion, if required). As the latter enables the accurate

waiting until a pattern is displayed before capturing it. If

registration of projectors into a 3D environment, it has the

temporal multiplexing-based projectors are used, such asadvantage that the devices can be easily integrated into the

DLP projectors, care has to be taken to use long enoughcontent generation pipeline of the installation.

exposure times in exact accordance with a projector’srefresh rate or some multiple of it to avoid color andintensity artifacts. Averaging multiple camera images of・Multi-Projector Blending and Shadow Removal:
Depending on the surface complexity, one usually tries

the same pattern helps to reduce these artifacts. As soon as

to overlap the contributions of multiple projectors, as it

multiple projectors are used to present dynamic content,

may simply not be possible to align multiple projectors so

their display refresh timings have to be synchronized withthat they do not overlap without any black gaps between

each other to avoid the occurrence of tearing artifacts.

27

28
92

特集

日本バーチャルリアリティ学会誌第 19 巻 2 号 2014 年 6 月

プロジェクションマッピング

While this is technically possible without any significant

In addition, these systems have to detect and report potential

problems, it increases the overall complexity and cost of theerrors such as hardware defects or other issues that restrict theirsystem.

functionality, such as occluded lines of sight or unexpected
lighting situations. Obviously, all of this has to be carried out not

Depending on the application, not all of the above mentioned

only efficiently, but also within a short period of time.

points are necessary to generate a satisfactory impression,
though focus should be kept on all of them when developing

The most crucial factors that have to be solved to generate

projection-based AR setups. Most of mentioned issues canrobust procams systems are listed as follows:

be solved manually through the use of specific software
calibration tools. Particularly when multiple projectors are・Robustness: Since theme park installations commonlyused in a complex setup, manual tools quickly lead to arun daily for many hours and are planned to run for several

cumbersome or even unfeasible process and cameras should

years or even decades, it is required that, independent of

be used for automatic system calibration.

any internal or external factors, a procams system be able
to provide desired image quality, even in unpredictable

Much research has been put into automatically solving thesituations. If, for whatever reason, satisfactory image

above-mentioned issues with different focuses on image

quality cannot be achieved, a system has to be able to detect

quality or processing speed depending on the application.

and report this without human intervention.

Methods for geometric calibration can be applied partiallyin real-time or without any pre-calibration. Blending maps
and photometric compensation images can also be calculated
for impressive image quality even if non-trivial projection
surfaces are used [7].
Despite all these resolutions to various issues, however,one major problem remains: can these fixes be successfully
applied in long-term installations operated and maintained by
non-expert users within environments that are not perfectly
controlled? How can such a system be set up so that highimage quality can be achieved reliably while at the same time
keeping calibration and maintenance times to a minimum?

Figure 3
Disney Research Zurich developed a method which uses projection tosuperimpose high frequency texture details onto an animatronic headto significantly enhance its appearance [6].

3．Challenges of Long-Term Projection Based AR
Installations
Several solutions for projection-based AR application calibration
issues have been developed, but due to their complexity, many・Accuracy: The requirements for spatial projection

require experts to accurately apply them when setting up aaccuracy depend significantly on the setup conditions.

system, adjusting parameters to adapt to noise and other factors

Obviously, a large-scale building projection that exists

in a given setting, and judging the accuracy of a final projection.

far from observers requires less accurate geometrical

Aside from these issues that rely on experts to correct, otherregistration than a projection of fine details on anissues have to be solved for long-term applications, such asanimatronic head [ 6 ]. Such conditions thus have to beentertainment attractions in Disney theme parks. At locations

carefully considered when setting up a system and,such as these, it typically cannot be expected that operational

even more important, attention has to be paid that theand maintenance personnel with expertise in computer visionvariance within consecutive recalibrations required forwill be present. Therefore, procams systems have to be able

hardware replacements, cleaning operations and otherto automatically calibrate themselves with little supervision.

28

特集

プロジェクションマッピング

JVRSJ Vol.19 No.2 June, 201429
93

system restructurations is kept at a relatively low level.

and operational costs as low as possible is a strong

As non-experts typically take care of technologies during

requirement in theme park installations. This factor has tooperational hours and maintenance, automated, self-

be carefully considered when designing new attractions withassessing calibration tools such as camera-based self-each of the above-mentioned issues in mind. It is obvious thatcalibration can help to assist them.

the more complex the system gets, the more potential sources

・Scalability: Lab installations typically deal with projectorsof error there are. Therefore, a major goal when designingthat are professionally set up for particular experiments in

attractions is to find the simplest solution for a system tocarefully controlled environments. In contrast to this, real-guarantee desired accuracy given its conceptual and temporalworld installations often involve constraints in terms of

constraints.

projector and camera placement, as well as may require
more projectors to generate desired pixel coverage— and4．Conclusions

all of this has to be considered while developing robust

Projection-based spatial AR is an effective and flexible way

calibration algorithms. Optimally, algorithms should not

to generate dynamic shading effects on complex structured

scale more than linearly in terms of processing time andsceneries with a quality that cannot be achieved by only

memory requirements. They should, however, enable

using a large number of standard light sources. This makes

parallel processing, and, even more important, employ

this technology quite suitable for scenic entertainment

parallel acquisition whenever possible. All cameras, forinstallations, such as those used by The Walt Disney Company

example, should be able to capture structured light patterns

in a multitude of their theme park attractions. Some of thefrom specific projectors in parallel, and, having alreadyattractions in which projectors are used to generate interactive

gathered some rough knowledge about the geometrical

game applications, immersive projections, and complex

setup, even non-influencing projectors can project patterns

lighting effects are shown in the figures throughout thisin parallel to speed up acquisition times.

document.

・Performance/speed: As the downtimes of everyday
projection systems should be kept to a minimumfor obvious reasons, calibration and compensation
algorithms are required to compute the results as fast
as possible. While optimizing calibration processing
times in non-real-time applications is not of essential
interest within research communities, the duration ofthese times is a crucial factor in theme park attractions.
Therefore, on the one hand, efficient algorithms haveto be implemented by excessively using CPU and GPU
parallelism whenever possible. On the other hand,

Figure 4“Goofy's Paint'n'Play” at Disneyland Tokyo is an interactive game in which
a self-calibrating projector-camera system is used to accurately projectvirtually shot paint and textures onto walls and furniture.

processing hardware has to guarantee high bandwidth
during acquisition times, as well as enough processing
power for computations. As most of the calculations at＊ The color version shown on the frontispiece

least linearly scale with the number of projectors used,hardware has to be carefully adapted to offer enough

The majority of the above-mentioned problems involved inprocessing resources for the given problem.

establishing a robust and reliable everyday procams installation

・Maintainability: Despite having solved for all of the

can today be resolved by using specialized software tools. The

above issues, the question remains as to whether a system

Walt Disney Company developed and still utilizes a procams

can be maintained in a reasonable amount of time andtoolbox framework that offers a set of flexible components

cost-efficiently. The deployment of new technologies isthat can be individually customized and combined to fit thecommonly constrained by the fact that keeping maintenanceparticular needs of any new installation. However, not every

29

30
94

特集

日本バーチャルリアリティ学会誌第 19 巻 2 号 2014 年 6 月

potential problem in projection-based AR has been solved for

プロジェクションマッピング

Stesin and Henry Fuchs, ACM Siggraph 1998

yet. The following gives a list of outstanding problems:[3] “IllumiRoom: Peripheral Projected Illusions for Interactive

・Dynamic projection surfaces: Projecting onto dynamic

Experiences.” Jones, B., Benko, H., Ofek, E. and Wilson,

objects adds a series of new challenges to projection-based

A. D. 2013. In Proceedings of the 31st Annual SIGCHIAR, such as real-time scene analysis, low-latency input andConference on Human Factors in Computing Systems

output systems, and projection adaptation onto potentially[4] “Projection-Based Augmented Reality in Disney Theme

non-rigidly deformable surfaces. It is without question that

Parks”, M. Mine, D. Rose, B. Yang, Jeroen. v. Baar, A.

these issues will significantly complicate a system’s design

Grundhöfer, IEEE Computer vol. 45, no. 7, 2012
[5] “Perceptual photometric seamlessness in projection based

and place much greater limitations on algorithm complexity

tiled displays”, A Majumder, R Stevens, ACM Transactions

and runtimes.

on Graphics（TOG）24（1）, pp. 118-139

・Low latency: Even when projecting onto static surfaces,interactive applications require short or even perceptually[6] “Augmenting Physical Avatars Using Projector Basedinstantaneous responses so not to distract users. This

Illumination”, A. Bernano, P. Brüschweiler, A. Grundhöfer, D.

requires not only a fast processing time, but also a low

Iwai, B. Bickel, and M. Gross, ACM Transactions on Graphics

latency hardware system to give immediate feedback.

（Siggraph Asia）, 2013, Hong Kong

・Complex surface materials: The projection onto

[7] “Practical non-linear Photometric Projector Compensation”,

complex materials generating non-diffuse lighting effects

A. Grundhöfer, CVPR workshop in computational Cameras
and Displays（CCD）, 2013

like subsurface scattering or glossy reflections requires amuch more detailed surface analysis to accurately model[8] “Radiometric Compensation through Inverse Light Transport”,

the occurring light transport, as well as to compensate

G Wetzstein, O Bimber, Pacific conference on computer

for these effects as best as possible given any physicalgraphics and applications, pp. 391-399

constraints. One recent approach to neutralize these effects
is shown in Figure 3, where fine details missing on the skin

【Biography】

of an animatronic figure are superimposed by optimized

Dr. Anselm Grundhöfer is a Senior Research Engineer at

projection images that compensate for subsurface scattering

Disney Research Zürich, Switzerland, where he works in Dr.

and defocus effects [6].

Paul Beardsley's computer vision group as technical lead of
the Procams Toolbox project. He graduated with a degree in

Overcoming these issues represents a major task for future

Media Systems Sciences in 2006 and obtained his doctorate

procams research. While several publications already showed

in Engineering in 2010 at Bauhaus University, Weimar,

promising results in incipient stages, there is still a long way

Germany. During his studies, he spent several months at theto go in terms of enabling everyday projection systems to

Takemura Lab at Osaka University, Japan.

consider and deal with these limitations, but solving theseAnselm's work is mainly focused on developing procams

issues would offer a completely new area of interactive

systems and assisting in their deployment, for example, in

projection-based applications. Hopefully researchers all over

Disney theme park attractions. Most recently, he is working

the world will help to make those procams system happen in

on multi-projector optimization methods to overcome several

the near future.

unresolved problems involved when projecting onto nontrivial surfaces. Aside from this, he also conducts research

Referenceon computer vision, AR, video processing, and display[1] “The Visual Computing of Projector-Camera Systems”, O. Bimber,technologies.

D. Iwai, G. Wetzstein, A. Grundhöfer, Eurographics Stateof the Art Report, 2007, Praque, Czech Republic[2] “The Office of the Future : A Unified Approach to ImageBased Modeling and Spatially Immersive Displays,”,Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev

30