Hindawi Publishing Corporation
EURASIP Journal on Advances in Signal Processing
Volume 2011, Article ID 572596, 17 pages
doi:10.1155/2011/572596

Research Article
ipProjector: A Vision-Based Framework for
Unsupervised Ubiquitous Object Augmentation Usinga Portable Projector-Camera System
Thitirat Siriborvornratanakul and Masanori Sugimoto
Interaction Technology Laboratory, Department of Electrical Engineering and Information Systems, The University of Tokyo,
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan
Correspondence should be addressed to Thitirat Siriborvornratanakul, thitirat@itl.t.u-tokyo.ac.jp
Received 13 July 2010; Revised 25 November 2010; Accepted 9 February 2011
Academic Editor: Karen Panetta
Copyright © 2011 T. Siriborvornratanakul and M. Sugimoto. This is an open access article distributed under the Creative
Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided theoriginal work is properly cited.
ipProjector stands for interactive portable projector which is a project trying to make a smart projection system while retaining
absolute portability of the devices and setups. In this paper, an ipProjector framework is proposed for unsupervised real-time
object augmentation applications. Scene-independent projector-camera geometric calibration is introduced so that the projectedaugmentation always appears in the desired geometry despite an unconstrained surface. Using nonintrusive projection, the system
can ensure correct vision-based analysis because external appearances of the environment and object (as seen by the camera) arehardly changed by the overlaid projected content. Finally, an unsupervised visual detector is used to guarantee robustness of thedetection regardless of lighting variations. Several image processing algorithms are applied in the framework, including image
warping using perspective transformation, color conversion using the nearest-neighbor search algorithm, and multiscale visual
detection. Based on a small-scale laboratory setting, experiments were conducted to evaluate the proposed approaches, and one
example was built to prove the proposed concept in object augmentation.

1. Introduction
Because of the trend in mobile projector phones, people havestarted to imagine using mobile projectors ubiquitously inthe same way as mobile phones. A good ubiquitous system isa matter of mobility and intelligence. While the mobility of a
projector has recently been successfully proposed, intelligent
mobile projection has not yet been accomplished in practice.
In this paper, we focus on a vision-based ubiquitous
projection system that is able to perform real-time object
augmentation in a robust manner. Only geometry aspect isconcerned, and our goal is to create a system that can project2-D augmentations at the desired position corresponding to
the target object in a 3-D environment.
Object augmentation using a projector has been proposed continuously in the past decade. This trend has
become more prominent with the recent proposal of the
SixthSense project [1] that uses a wearable projector toaugment the physical world with virtual information projected from a projector. The vision-based projector-camera(i.e., pro-cam) system is often the choice for researchers inthis field because it allows object augmentation everywhere
without relying on separate tracking hardware, and a user
can interact directly with the projection.
There are two main problems encountered when making
a pro-cam system aware of its surrounding objects, while stillretaining ubiquity. First, ubiquitous projection implies use in
an unknown environment that has not been prepared. However, geometrically calibrating the pro-cam system requires
creating a strong geometric relationship between projector,
camera, and surface. Hence, the unconstrained surface
becomes an important factor that makes the calibration
become very diﬃcult. The second problem, which we call“projective interference”, is caused by the projected contents
being captured by the camera. Suppose that the camera is
working in the visible spectrum at a standard shutter speed,

2
the projected contents overlaid on the environment are also
seen by the camera. For vision-based systems, this leads tofalse detections of the environment and object unless thesystem has a way to deal with the interference eﬀectively.
Apart from the two mentioned problems, there is no
doubt that the key success of object augmentation is to detect
the target object correctly in the first place. This meansthat a visual object detector used in the system must be
robust enough in a real-world scenario. However, there is afundamental conflict using projection and visual detection
together in the visible light spectrum. Namely, projection
requires a dark environment for better visualization of the
projected images on the surface whereas the visual object
detection algorithm usually prefers input images captured ina bright environment so that the sharp details of objects are
obtained with few additive noises. Combined with the factthat visual appearances of the object can be changed easily
by the environment lightings, the problem of visual object
detection becomes more challenging for a ubiquitous procam system.
All proposed approaches are designed to be robust to
unknown geometries, textures, and variations of lighting in areal-world scenario of ubiquitous projection. Contributionsof this paper can be applied to other vision-based pro-cam
systems to ensure robust geometry-based object augmentation whose accuracies are barely aﬀected by environment
lightings or appearances of the surface and object. Flexibility
is another advantage of this framework. Any oﬀ-the-shelfdigital light processing (DLP) projector can be used to implement a system following this framework. A representative
detector used in this paper can also be replaced by another visual object detector with no internal modification
to the detector. Finally, this framework supports integrationof other vision algorithms (e.g., tracking algorithm, gesturerecognition, adaptive learning, etc.), enabling realistic object
augmentation to achieve in an actual application.
The rest of this paper is laid out as follows. Section 2
explains recent advances in object augmentation using a
projector and then discusses related research in the threeproblem areas as mentioned earlier. Section 3 introduces the
proposed framework and the overall system configuration.
Section 4 presents a self-contained and portable pro-cam
design whose geometric calibration can be achieved on bothplanar and nonplanar surfaces with no prior knowledge of
their geometry. Section 5 describes problems of variationof lighting regarding the proposed design and presents oursolution of the unsupervised visual object detection. Section 6 then explains the nonintrusive projection techniqueusing a DLP projector to solve the projective interference
problem. A simple program is built and shown in Section 7 todemonstrate the overall concept of the proposed framework.
In Section 8, performances of the proposed approaches areexperimentally confirmed. Finally, Section 9 concludes thisarticle.

2. Related WorkAs mentioned, our achievement is related to three typesof problems. In the following sections, we explain recent

EURASIP Journal on Advances in Signal Processing
advances in object augmentation using a projector and thendiscuss related research in the three areas.
2.1. Object Augmentation by Projection. So far, researchers in
the field of ubiquitous pro-cam systems focus on developing
new interactive techniques but pay scant attention to thefundamental problems (as mentioned in Section 1) in areal-world scenario. Previous works of augmenting objects
by projection tend to neglect or simplify these problems
with various alternatives that do not meet the goal of being
ubiquitous. Paper-based fiducials are attached to the target
object in SixthSense [1], iLamps [2], and Shelf Torchlight[3]. The rectangular handheld display screen is recognized
and tracked by its black border in [4] and by four lightsensors attached to its corners in [5]. Three infrared LEDs
forming a triangular shape are placed on the robot allowing
robot manipulation using a mobile projector in CoGAME[6]. So far, it can be said that using a visual marker is themost popular solution for a ubiquitous pro-cam system. It
significantly simplifies the problems of a ubiquitous pro-cam
system, enabling creation of a system that can extract only
the desired information and disregard the others (including
information suﬀering from the projective interference).
Nevertheless, these works are not ideally ubiquitous yet,because they either modify or engineer external appearancesof the object or surface beforehand.
Other works that do not incorporate with alternation of
object appearances are shown in [7–9]. The work of Kanbara
et al. [7] uses a projector to project invisible markers which
can only be seen by the specific camera. An environmentaware display system proposed by [8] embeds the imperceptible stripe pattern into the normal projection, andreal-time object detection is achieved without any marker.
A commercial 3-D tracking unit is used in [9], enabling
physical object annotation to be performed based on the 3-D
position tracking strategy. All these works can solve the earlyproblem of using visual markers by not modifying external
appearances of the object (as seen by users). However, noneof them retains absolute ubiquity and portability in theirimplementation [7] requires the invisible markers to be
projected steadily onto a wall or ceiling, and moving the
projector or object is not allowed. The system in [8] is limited
to fixed projectors and fixed cameras mounted on a ceiling,
and projection surfaces are restricted to flat table surfaces
whose distance to the ceiling is unchanged. In [9], one
stationary camera is needed in the workspace for assisting the
3-D tracking unit. The cooperative augmentation proposedby [10] seems to be the appropriate solution for real-time
object augmentation in ubiquitous projection. Their procam system dynamically configures its visual object detection
based on four diﬀerent detection algorithms that ensure
detection coverage in a real-world scenario. Nonetheless,this system relies on an assumption of the smart object
where object-model knowledge must be embedded during
manufacture.
Unlike these systems, our proposed framework anddesign allow creating a markerless system that is truly selfcontained and well calibrated despite the unconstrained
environment and dynamic objects. Besides, the framework

EURASIP Journal on Advances in Signal Processing
is automatic and involves no user feedback, training, or
supervision during online execution.
2.2. Pro-Cam Geometric Calibration. So far, researchers have
tried proposing a new approach to calibrating the pro-cam
system precisely on diﬀerent surfaces. When a projector anda camera are rigidly fixed to each other, some have assumed
that the geometric registration between them is roughly
constant [4]. However, as the angle of the projector moves
from the perpendicular, or as a surface becomes nonplanar,
this approach will no longer guarantee good geometricregistration. In Map TorchLight [11], a mobile phone is
fixed to a mobile projector, and a known paper-based map(containing visible black dots arranged in a regular grid) isused to assist real-time geometric calibration. Projecting aknown pattern onto a surface is a classical approach to solve
this problem that gives precise calibrations for both planar
surfaces [12–14] and nonplanar surfaces [2, 15, 16]. However, the computational cost is high for a complex surface,
and the pattern must be reprojected when any componentof the system (e.g., a projector, camera, or surface) moves. A
similar approach is applied in the catadioptric projectors [17]whose projected light is refracted and reflected by refractors
and reflectors, respectively. Geometric registration between
the two devices is obtained by projecting a series of known
patterns and allowing the camera to sense them. A realtime approach that does not interrupt normal projection
was proposed in [18] by attaching four laser pens to a
handheld pro-cam device. Although detecting bright laser
points sounds easy, locating small laser points in a complex
camera image is still diﬃcult in practice. In [19], Johnson
and Fuchs proposed a real-time approach that does notinterrupt the normal projection, requires no fixed marker,
and can be applied to a complex surface. By matching
feature points found in the projected image and the predicted
captured image, the pose of the projector is tracked and thecalibration is achieved in real time. However, the camera is
stationary and separated from the projector in their system.
Unlike these systems, our proposed calibration is truly
ubiquitous. It can perform the calibration in real time
without interrupting the normal projection or requiring any
stationary device in the workspace. Besides, both planar andnonplanar surface can be used as the projection surface withno prior knowledge of their geometry.
2.3. Nonintrusive Projection. Nonintrusive projection can beseen as an inverse problem of the embedded imperceptible
pattern projection whose projected pattern is invisible tousers and normal cameras but visible to the calibrated camera. There are many solutions proposed for the imperceptible
pattern projection but few for the nonintrusive projection.
Fortunately, some solutions can be shared between thesetwo problems with slight modifications. In the oﬃce of the
future [20], structured light can be embedded into a DLP
projector by making significant changes to the projection
hardware. However, this implementation is impossible unless
it is incorporated into the design of the projector or fullaccess to the projection hardware is available. In [21–23], a3code image is projected at high speed with its neutralized
image which integrates the coded patterns invisibly due tolimitations of the human visual system. According to these
papers, projecting and capturing at 120 Hz can guarantee
a hidden code, but commonly available projectors usually
perform projections at a maximum rate of 87 Hz. Theseprevious works try to invisibly embed the binary pattern tothe normal projection. By changing the embedded pattern
to the uniform white image (the reason regarding this will
be explained at the end of Section 6.2), some of them canperform nonintrusive projection.
For this paper, we apply an approach based on the DLP
characteristics. Recently, internal characteristics of a DLP
projector have received attentions from research communities. In [24, 25], two approaches utilizing the internal
characteristics of a DLP projector are proposed. First, thedithered illumination pattern corresponding to the DMD
chip (which operates at 10,000 Hz) is observed and utilizedusing a very high-speed camera (whose maximum speed is3000 fps). Second, characteristics of the color wheel (whichrotate at 120 Hz) is investigated and used by a camera
with slower capturing speed. Our nonintrusive projection
is based on the color wheel solution and has three main
advantages: it requires no internal change to the projector
or the camera, it can be applied to any oﬀ-the-shelf DLP
projector, and it supports embedded variable light patterns(i.e., imperceptible pattern projection) in the future without
further hardware modifications.
2.4. Robust Visual Object Detection. Variations in lighting are
the classic problems that have been discussed in computer
vision communities for decades, particularly for surveillance,
cognitive, or mobile systems where a specific environment
cannot be assumed. Recent researches of robust object detection have focused on applying visual learning or statisticalmodeling so that a specific object or a specific class of objects
can be detected in all possible scenarios. For model-based
approaches, many works try to build a generic model ofthe target object applicable for all lighting scenarios. For
example, face detection under variant illumination video
sequences is solved in [26] by building a skin-tone model andsegmenting each video frame into skin and nonskin regions.
The idea of the model-based approach is to train the systemusing an exemplar set representing appearances of the target
object under all scenarios. Two drawbacks of this approach
are that the exemplar set may not be available or adequate
for some applications, and this approach does not support
detecting an unknown object. Instead of building a modelof the target object, researches in [27, 28] reduce sensitivity
to variations of lighting by modeling the background andthen using background subtraction to extract the foreground(including the target object). Although these systems are
adaptive to variations of lighting and allow detection of an
unknown object, they assume uses in the specific backgroundonly. A similar idea is applied in [29] for a cognitive
mobile robot system. Assume that the robot acts in aclosed environment and repeatedly returns to the sameplace, the statistical model of lighting behaviors is built,
enabling the robot to predict lighting of the environment at a

4

EURASIP Journal on Advances in Signal Processing

Object detection: locate the projector’s frustum: detect target objects

Response generation: create interactive responses

Geometric calibration 2: image warping

Projected image

Projector
coordinates

Nonintrusive projection

Geometric calibration 1

Camera coordinates

Camera image

Figure 1: Procedural flow of the proposed framework during theonline execution.

VGA splitter

DLP projector
Beam splitter
CameraFigure 2: Overall system configuration.

specific location. Using a model limits the system to perform
detection only where the model is available as discussed. To
solve this problem, a robust adaptive object detection that
does not rely on a model created oﬄine is shown in [30].
By using an online learning algorithm, the system can adapt
to changing environments with no need for the predefinedbackground model. Nevertheless, this system also requires
environment-specific information to reduce the number offalse alarms. Therefore, it cannot be applied in a portablesystem.
It can be seen that detecting an unknown object invariations of lighting is still a challenging problem incomputer vision. Fortunately, our proposed system is barely
aﬀected by the unpredictable environment lighting because
the projector’s red light is the only light source illuminating
the environment as seen by the camera (as detailed in
Section 6). Therefore, variations of lighting can be simply
explained as an inverse relationship of the distance fromthe projector to the object (please refer to Figure 5(a)). In
other words, variations of lighting in our system are notas complex as the related works in this field. Hence, we
decided to apply the multiscale concept to ensure detection
coverage in our system. Other reasons that we chose thismultiscale concept are as follows: (1) it requires no internal
modification in the base detector, (2) it is flexible because
the base detector can be replaced easily with another visual
detector to match individual requirements, and (3) increases
in computation can be easily oﬀset by the recent growth ofmulticore processors and parallel programming languages.
In this paper, we propose a complete vision-based
framework and a portable pro-cam design for ubiquitous
object augmentation applications. The goal is to accomplish
a pro-cam system that (1) retains absolute ubiquity despite
the projection environment, (2) allows projecting real-time
augmentation everywhere, and (3) is able to reliably detectobjects inside the projector’s frustum without supervision.
The framework consists of three main parts as the solutions
for the two mentioned problems and the robust visual object
detection. Experiments were conducted to evaluate the proposed approaches in a small-scale laboratory setting. Actual
object augmentation application has not been implemented

in this paper, and only one program projecting simple
augmentation is shown as the proof of concept.

3. Framework Overview and
System Configuration
In this section, we briefly explain the proposed frameworkand the system configuration used in the paper. As illustrated
in Figure 1, our proposed framework for real-time object
augmentation consists of five main tasks (written inside
the rounded rectangles) during the online execution. First,
the projection frustum appearing inside a camera image islocated (Geometric calibration 1). Then, the target object
existing inside the frustum area is detected (Object detection).
After that, interactive augmentation is calculated for eachdetected object (Response generation). A nonintrusive projection is applied in this step to guarantee that all augmentations
are drawn using the correct colors. Up to this point, allcalculations are done in camera coordinates. Finally, anothercalibration is performed to convert every projection content
to projector coordinates (Geometric calibration 2) so that it
appears in our desired geometry on the actual surface; thistask is sometimes referred to as image warping.
Information regarding geometric calibration 1 and 2,
object detection, and nonintrusive projection are shown in
Sections 4, 5, and 6, respectively. The response generationis performed using diﬀerent algorithms (depending on
the application), so a detailed explanation of it is notconsidered here. Following this framework, the system is able
to detect the target object and project the corresponding
augmentation in our desired geometry. Note that thisframework focuses only on the fundamental tasks based on
the frame-by-frame analysis strategy. Additional algorithms
can be added before the response generation step, enabling
accurate detection or realistic augmentation in a realworld application. For example, adding a tracking algorithm
will help a system to augment with the target object(s)in a continuous manner, using gesture recognition will
support direct user interaction with the projected image, andapplying object recognition will allow recognizing a specificEURASIP Journal on Advances in Signal Processing

Initial setup

5: fix the projector, camera, and beam splitterat the appropriate positions: collect a set of corresponding pixels between
camera and projector coordinates: pairs of known correspondences: compute geometric mapping by perspective
Find pro-cam transformation transformation: RANSAC for estimating the transformation: homography matrix HCP(converting from camera to projector coordinates)
Collect correspondences

Finish geometric calibration (oﬄine): homography matrix HCP
Geometric calibration 1(online)Geometric calibration 2: use [HCP ]−1 to locate the projector
frustum in the captured image: use HCP to warp the interactive response
image to projector coordinates

Figure 3: Procedural flow of the proposed scene-independent pro-cam geometric calibration during oﬄine and online execution.

object or a specific class of objects whose appearances match
requirements.
Our system configuration is shown in Figure 2. A camera,
beam splitter and DLP projector are fixed firmly on awooden base forming a single self-contained device (seeSection 4 for details) whereas a VGA splitter is added forpro-cam synchronization (see Section 6 for details). The
device is completely self-contained and portable; however,re-engineering is required to make it more compact for
ubiquitous uses.

4. Scene-Independent
Pro-Cam Geometric Calibration
Any pro-cam system is related to three pairs of geometric calibration: projector-camera, projector-surface, and camerasurface calibration. For an application such as the automatic
keystone correction (using a vision-based solution), all threecalibrations have to be performed concurrently. However,for projecting 2-D augmentations at the desired location
regardless of the 3-D shapes of the surface, only pro-cam
calibration is adequate. Geometric mapping between cameraand projector coordinates is used to find corresponding
positions between the two coordinate systems and to project
images back to desired locations on an actual surface.
These two tasks refer to the geometric calibration 1 and 2,
respectively, as illustrated in Figure 1.
Pro-cam geometric calibration is one of the problems of
most concern in a ubiquitous projection system. Because,
although the projector and the camera are fixed together,poses and orientations of the surface (compared to the two
devices) are still unknown. On a planar surface, when theangle of a projector moves away from perpendicular, thegeometric mapping changes. On a nonplanar surface, there
is an additional problem as 3-D shapes of the surface create
parallax eﬀects between projector and camera coordinates.
This nonoverlapping fields of view between the projector
and the camera may make geometric mapping impossible

in some positions. Using a 3-D scanner to render the 3-Dmodel of the surface is an ideal solution that tracks relative
positions and orientations of the projector compared tothe surface in real time. However, this is still diﬃcult toperform accurately in real time using a portable device inan unknown environment, particularly when the reference
device (i.e., projector and camera) and the target object are
not stationary.
Figure 3 shows the summary of our calibration approach.
The oﬄine processes (shown inside the dashed-line rectangle) consist of setting up the beam splitter and computing
geometric mapping between projector and camera coordinates. The outcome of these oﬄine processes is the homography matrix representing the geometric transformation fromcamera coordinates to projector coordinates and vice versa.
During the online execution, there is no need to recalculate
this matrix, and the geometric calibration can be achieved
regardless of 3-D geometry of the projection surface.
In the following sections, we explain in detail how toachieve the mentioned geometric calibration using a single
self-contained device. Section 4.1 (which refers to the initial
setup step in Figure 3) shows a colocating pro-cam design
whose geometric calibration is considered to be independentof the surface. Section 4.2 (which refers to the rest of theoﬄine processes and the online processes written in Figure 3)
then describes how to use perspective transformation model
to calculate pro-cam geometric mapping based on the designpresented in Section 4.1.
4.1. Colocating the Projector and Camera. Previous researches
on mobile projection have assumed projections on a uniform
planar surface, which is the most primitive projection surface. The reason behind this assumption is to eliminate any
eﬀects due to the surface geometry. In previous researches,
although a camera was firmly fixed to a mobile projector,
their optical axes were not exactly aligned. Therefore,
geometric conversions between them were aﬀected by thegeometry of the projection surface. Using 3-D surface
rendering is an indirect solution of this problem; however,6

EURASIP Journal on Advances in Signal Processing

(a)

(b)

(c)

(d)

Figure 4: Comparison between the noncolocated (b, d) and colocated (a, c) pro-cam devices. (a, b) show comparisons on a planar surface.
(c, d) show comparisons on a curved surface.

it is still diﬃcult to perform in real time for a mobile system.
To achieve a robust pro-cam system, we use a straightforward
solution by colocating the projector and camera so that thesurface factor is eliminated from the calibration process. As
a result, geometric conversions between the two coordinateshardly change and can be said as being independent of thesurface.
A beam splitter is an optical device that reflects half of theincoming light and transmits the other half. There are few
researches concerning a beam splitter for pro-cam systems.
Our design proposed in this section was inspired by thework of Fujii et al. [31] which briefly described the ideaof scene-independent geometric calibration using a platebeam splitter attached to an oﬀ-the-shelf projector. Boththeir research and ours operate in the visible light spectrum;
however, the camera settings are completely diﬀerent. In
their research, a camera uses a standard shutter speed
and works independently to a projector. In our research(as described later in Section 6), the camera is accurately
synchronized with the DLP projector and its shutter isopened for only 0.55 ms.
By colocating the two devices using a beam splitter(as shown in Figure 2), the camera sees exactly what a
projector is projecting, and 3-D geometry of the surface
hardly aﬀects the geometric conversions or causes parallax
between projector and camera coordinates. An example
is shown in Figure 4. There is parallax between the twocoordinates when using a noncolocated pro-cam device on
a nonplanar surface (Figure 4(d)) so that some parts of the
projected surface cannot be seen by the camera. Comparedwith Figure 4(c) of the colocating design, the shape of thesurface does not cause significant distortion of the projected
pattern and shadowing is also reduced.
In summary, this design is very useful for geometriccalibration that requires high conversion accuracy between
the two coordinates. An example is shown in [24] where
the colocating design is used for direct-global separation inorder to accurately map camera pixels to the corresponding
projected intensities. For ubiquitous projection, this design
is very interesting because (1) it requires no external device
stationary in the workspace, (2) precise geometric calibrationis possible on any surface, (3) no additional computations are
required, (4) it ensures that any surface visible to the camera
can be projected upon, and (5) it eliminates shadows incamera images. Nevertheless, this approach is not suitable forpro-cam applications that utilize distortions of the projected
images on the surface. For example, as occurs with 3-Dreconstruction using projected structured light.
4.2. Perspective Transformation Model. In this paper, geometric mapping between camera and projector coordinates
is computed by perspective transformation [32] whose
computation eﬀort is lighter than Euclidean calculation,
suitable for real-time interactive applications. Based on
the fact that all points seen by the camera lay on some
unknown plane, the perspective transformation between thetwo coordinates can be established by a 3 × 3 homographymatrix. Suppose that (X, Y) is a pixel in projector coordinates
whose corresponding pixel in camera coordinates is (x, y),
the perspective transformation from (x, y) to (X, Y) can be7expressed with eight degrees of freedom in homogeneous
coordinates as
⎛

Xw

⎞

⎛

h1 h2 h3

⎞ ⎛ ⎞

x

⎜
⎟
⎜
⎟ ⎜ ⎟
⎜ Y w ⎟ = ⎜ h4 h5 h6 ⎟ · ⎜ y ⎟ ,
⎝
⎠
⎝
⎠ ⎝ ⎠

w

h7 h8 h9(1)1CP = (h1 · · · h9 )T is constrained by condition
where HCP | = 1 and can be computed from four corresponding|Hpixels between the two coordinates (four correspondences
ensure that no three points is collinear). When there are
more than four corresponding pixels found between the twocoordinates (δ > 4 in (2)), the RANSAC method is appliedCP in the following equation:for estimating the values of H⎛

X1 w X2 w · · · Xδ w

⎞ ⎛h1 h2 h3

⎞ ⎛x1 x2 · · · xδLuminance(as seen by the camera)

EURASIP Journal on Advances in Signal Processing
140
120
100
80
60
40
20
0
20 30 40 50 60 70 80 90 100 110 120 130

Setup 1 120 100 91.6 85.9 75.3 60.1 46.8 37.8 31.4 25.9 22.1 19.3
Setup 2 96.1 87.8 71.4 50.8 37.1 28.6 22.5 18.4 15.6 13.6 11.8 10.3
Setup 3 83.1 56.4 36.6 25.8 19.4 15.4 12.9 11 9.71 8.82 7.88 7.22
Distance from the projector to the planar surface (cm)
(a)

⎞

⎜
⎟ ⎜
⎟ ⎜
⎟
⎜Y1 w Y2 w · · · Yδ w ⎟=⎜h4 h5 h6 ⎟ · ⎜ y1 y2 · · · yδ ⎟.
⎝
⎠ ⎝
⎠ ⎝
⎠
w
w ··· wh7 h8 h9
1 1 ··· 1

Setup 1

Setup 2

Setup 3(2)
During the oﬄine processes, the system collects at least
four corresponding pixels between the two coordinates (e.g.,by projecting a known pattern on the surface) and uses themCP values following (2). After that, geometricto compute H
mapping from any (x, y) to (X, Y) and vice versa is achieved
1
CP and [HCP ] , respectively. Combined with theusing H
colocating pro-cam design (presented in Section 4.1) whosegeometric transformation is not aﬀected by the geometryof the surface, recomputation of this homography matrix is
not necessary if there is no change in the relative positions
or orientations among the projector, camera, and beam
splitter. In this way, scene-independent geometric calibrationis accomplished while ubiquity of the device is retained.

5. Multiscale Visual Object Detection
Because of the colocating design shown in Section 4 and theshort-exposure setting as described later in Section 6, thefundamental lighting diﬃculties of our vision-based system
are significantly underlined. Available light as seen by thecamera is very limited, and the projector’s red light is the
only light source illuminating the projection surface. Smallchanges in distance from the projector to the object or slight
depth variation of the surface can result in considerablyincreasing or decreasing the amount of light as seen by thecamera.
Figure 5 shows luminance of the projection as seen by
the camera in three setups, namely, Setup 1 for direct camera
capture with no beam splitter, Setup 2 for direct camera
capture with the beam splitter in front of the projector, andSetup 3 for our pro-cam setup using indirect camera capture
via the beam splitter. Using the camera setting as written
in Section 6.1, we constantly projected a uniform red image
onto a whiteboard located at diﬀerent distances and allowed
the camera to sense these red projections in the three setups.
According to the result shown in Figure 5(a), it is clear thatthe amount of light seen by the camera in our setup is(b)

Figure 5: (a) Luminance values seen by the camera at diﬀerent
distances to the surface in three diﬀerent setups. Note that the
maximum possible luminance here is 255. (b) The three setups usedin the experiment.

very limited and sharply decreased with the increase of the
distance. Note that the distance written in this context refers
to a distance from the planar surface to the front edge of thewooden base (as shown in Figure 2).
This significant decrease of luminance makes a visualobject detection become diﬃcult for a conventional standalone detection algorithm, as the detector tends to misssome true objects (a.k.a. undersegmentation) or include false
information belonging to noises (a.k.a. oversegmentation).
Considering a single input image, it is always possible toachieve a good detection result by fine-tuning parameters ofthe detector. However, adjusting these parameters manuallyduring real-time object augmentation is not preferred inour system. Therefore, we decided to use the multiscaleconcept to compensate the oversegmented result with theundersegmented result and to ensure that acceptable detection results will always be achieved with a single set ofpredefined parameters, regardless of diﬀerent or limited
lighting conditions.
In the following sections, we explain in detail how
to develop our proposed multiscale detection. Section 5.1
explains the overall concept of the proposed detection
scheme. Section 5.2 presents additional information regarding appropriate image simplification filters for the proposed
detection and the proposed pro-cam design.
Unlike color and texture, edges generally present across
any type of visual content. Hence, Canny edge detection
provided by the OpenCV library [33] is used in the following
sections as a representative base detector where, λ1 , λ2 , and σrepresent values of two Canny thresholds and aperture size,

8

EURASIP Journal on Advances in Signal Processing
Input image (RGB)

Input image (RGB)

Color conversion

Color conversion: luminance component: luminance component···

Scale 2Multiscale detection

Multiscale detection···

Scale 1

Scale N···

Base detector
×1

···

×2

+

×N

Image simplification
with minimum strength

Image simplification···

Scale 1···

Scale 2

Scale N···

Image simplification
with maximum strength

Histogram equalization

Histogram equalization

Base detector

×1: individual detection
output (binary)

···

×2

+

Scale map (grayscale) :×N: individual detection
output (binary)

Scale map (grayscale) :Thresholding

Thresholding

Final detection output (binary)

Final detection output (binary)

(a)

(b)

Figure 6: Procedural flows of the proposed multiscale visual detection scheme using sequential (a) and parallel (b) implementation.

Parallel multiscale

Sequential multiscale

Cameraimage
Scale 1

Scale 2

Histogram-equalized
luminance image
Scale 3

Scale 4

Scale map2

1
2

Scale 1

Scale 2

Scale 33

1
3

Scale 4

Scale map
5

4
5

6

4

6

Figure 7: Comparison between the sequential and parallel multiscale detection. Gaussian smooth filter and Canny edge detector are used as
the simplification filter and base detector, respectively.

respectively. Note that all input images used in these sections
are unsigned 8-bit RGB images. We strongly recommend thereader to look at the digital copy of these sections becausesome fine details cannot be seen when printed onto a paper.

5.1. Overall Concept. An overview of the investigated detection scheme is shown in Figure 6. It starts by converting
the image to YUV format and using only the Y (luminance) component in further calculations. This is related

EURASIP Journal on Advances in Signal Processing
to the observation that human eyes are far more sensitive
to luminance than color. Then, histogram equalization isapplied to enhance visibility and to standardize the input
image so that it is ready for the next multiscale detection
step. From our experiments, using only the multiscaleconcept (explained in the next paragraph) is not enough toachieve an unsupervised detection and adjusting parametersof the base detector is still required when dealing with
diﬀerently illuminated input images. Because our input
images (as well as other pro-cam systems operating in adark environment) suﬀer from insuﬃcient amounts of light,
using histogram equalization is a straightforward solution to
improve the overall appearance of the images, as shown in
Figure 7.
Generally, the multiscale detection refers to a hierarchy of detections where an original image is simplified
through iterations of a simplification filter as illustrated
in Figure 6(a). The idea is to use an image simplification
filter (with constant parameters) to decrease iteratively “toodetailed” information from the original image so that only
the information belonging to the true objects survives in thelast scale. By applying the base detector with the constant setof parameters to the simplified image in each scale, we can
combine the detection result from all scales to a single scalemap output using a weighted summation. Suppose that N
is the number of scales and i is an index of each scale, inthis paper, i equals to 1 refers to the highest scale (maximum
detail but least importance), i equals to N refers to the lowest
scale (minimum detail but highest importance), and theweight assigned to each scale is equal to i. Consequently, thehigh magnitude (black) in the combined scale map reflects
the high probability that this pixel belongs to the true edges,
as shown in Figure 7.
Nevertheless, using the sequential implementationmeans that the total number of scales is the key factor
to determine the computational time of the detection.
Considering recent growth in multicore processors andparallel programming languages, we changed the sequential
implementation to the equivalent parallel implementation
as illustrated in Figure 6(b). The histogram-equalized image
is distributed to all scales simultaneously, allowing multiple
simplifications to be performed in a parallel manner. Insteadof using the same simplification parameters iteratively as
the sequential implementation, our parallel implementation
increases the simplification eﬀect by directly enlarging theparameters that control the strength of simplification in eachscale. The experimental results presented in Figure 7 showthat the parallel implementation oﬀers similar scale-map
outcomes compared with the sequential implementation. In
this way, speed of the detection can be improved significantlywith few modifications.
After the multiscale detection step, the scale map image(grayscale) is converted to a binary image using a fixed
threshold value so that low-magnitude pixels, which usuallybelong to noises, are eliminated from the final detection
result. The investigated detection scheme is automatic andinvolves no user feedback, training, or supervision during
online execution. By simply replacing the representative
detector used in this section (Canny edge detector) with9a preferred visual detector, various object detections can beachieved for further use in object augmentation. The proposed detection scheme can also be applied to other pro-cam
systems regardless of our colocating and nonintrusive design.
However, some parameters may need to be reconsideredoﬄine to ensure that the simplification strength and thedetector’s sensitivity are set appropriately.
5.2. Image Simplification Filter. In addition to the detection
scheme mentioned in Section 5.1, there is an additional
issue of increasing textural noises caused by the histogram
equalization. The textural noise in this context refers to
textures that are hardly recognized in an original image
but become noticeable when strongly enhancing the image
with the histogram equalization. The more the histogramis changed, the clearer textural noises become and canresult in false positive visual detection. Therefore, using anappropriate filter to simplify the image is important to ensurethat these textural noises are not emphasized in the detection
output. For this section, we experimented on five smoothfilters: mean shift, conventional bilateral, real-time O(1)bilateral [34], median, and Gaussian filters. Meanshift andbilateral filters are edge-preserving smooth filters that oﬀer
“posterized” or “cartoonized” eﬀects over an input image.
Median and Gaussian filters are well-known smooth filters
usually used to reduce noises in an image.
Assuming that parameters of each filter are adjusted sothat the best detection result is achieved, Figure 8 shows
the scale maps obtained by applying the five filters over thesame histogram-equalized image in the parallel multiscalemanner. The number of scales (N) is 4, and the parametersof the base Canny edge detector are set to λ1 = 127, λ2 =1000 and σ = 5 in all experiments. From Figure 8, it
is obvious that the meanshift and median smooth filtersemphasize the textural noises in the scale maps whereas theconventional bilateral and Gaussian smooth filters oﬀer thecleanest and nicest edges. Compared with the conventional
bilateral filter, the real-time O(1) bilateral filter [34] canequally filter out the textural noises, but does better inpreserving edges and enhancing image contrast (the result
regarding this issue is not shown in this paper). However, its
results are distracted by noise pixels inside the object so that
continuous contours of an object are barely achieved. Notethat only the area inside the projector frustum is considered
here.
Furthermore, we examined the computational time usedby each filter. Under the same constraints, the computation
time used per one smoothing execution (without iteration)is 30, 63, 65, 18, and 15 ms for meanshift, conventional
bilateral, real-time O(1) bilateral [34], median, and Gaussian
filters, respectively. Because our input image here requires
strong smoothing eﬀects to reduce the textural noises causedby the histogram equalization, it costs much time in thesimplification processes. When dealing with other scenariosof less severe light conditions (not too dark), reducing
strength of the simplification filter is allowed and lesscomputation is required. Nevertheless, considering the time
as well as the resulting scale map, the Gaussian smoothfilter is the best choice that is not too sensitive to textural

10

EURASIP Journal on Advances in Signal Processing

1

2

Conventional bilateral filter

Histogram-equalized image Mean-shift filter

3

Real-time bilateral filter

54Median smooth filter

Gaussian smooth filter

(a)

1

2

3

4

5

(b)

Figure 8: The scale maps of the multiscale Canny edge detection (N = 4) using diﬀerent simplification filters. Top left is the histogramequalized image before the multiscale detection is applied.

noises, provides nice contours of the true objects, and uses
reasonable computational times.

6. Nonintrusive Projection
In order to achieve smart ubiquitous projection, the projection environment must be aware and recognized in real timeso that projected augmentations can be generated in the mostappropriate manner. Real-time environment analysis is not
new in research communities but, it is not straightforward
for a real-time projection system. In vision-based systems,
it is still diﬃcult to diﬀerentiate projected contents from
actual objects in camera images, particularly when the
projection surface is not planar and uniformly colored.
This problem is hardly mentioned in the work of previous
researchers investigating mobile projectors. Previously, thefocus was on the projection functions or the use of markers
to easily extract environmental information (as discussed
earlier in Section 2). In the electronic sticky notes andprojected augmented reality approaches proposed in [18],environment analysis is performed to detect an object inthe environment whose appearances match the preregisteredvalues. The interesting point here is that they seem to avoid
projecting anything over the target object because projected
contents superimposed on the target object will aﬀect thevisual matching process. In [3, 35], projecting over the target
object is allowed but the projected information is reduced
to a minimum to avoid significant changes in the object
appearances as seen by the camera.
Generally, there are two alternatives to deal with projective interference so that environment analysis and projection
can be performed simultaneously. One alternative createsa system that can be aware of all projected content at all

times. Handling projected contents in camera images is not
so diﬃcult if they do not overlap with physical objects.
The SixthSense project [1] utilizes this concept and keeps
tracking the projected content so that the system can handle
it correctly. This alternative certainly requires additional
computation for tracking and handling with the projected
content. However, as the projection surface contains many
details or textures, tracking the overlaid projected content
and handling with it should become much more diﬃcult.
The other alternative is to prevent the projected content
from being seen by the camera in the first place. The
straightforward approach to achieve this is to observe theenvironment using a sensing device insensitive to light (e.g.,
depth camera, laser scanner). For this paper, we selected thesecond alternative in order not to risk system ineﬃciencies
due to the surface complexity. The light sensing camera
is used but in a specific setting, allowing the projected
content to be invisible to the camera. Our approach is called“nonintrusive projection” which refers to the projection
approach whose projected contents are visible to humans andnormal cameras but invisible to the synchronized camera,
as an example shown in Figure 9(E). Therefore, the visionbased system can avoid projective interference (as seen by thecamera) that may lead to an incorrect environment analysis
like that shown in Figure 9(C).
Figure 10 shows the summary of our nonintrusive
projection approach. The oﬄine processes (shown inside thedashed-line rectangle) consist of synchronizing the projector
and the camera, analyzing the color wheel characteristics ofa DLP projector, and classifying the RGB color space into
three categories. The outcomes of these oﬄine processes are
the specific camera settings and three sets of classified colors.
During the online execution, only the color conversion is

EURASIP Journal on Advances in Signal Processing

G

A

B

C

D

E

F

11

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.510
(ms)
(a)
255

All these contents should
be invisible to the camera

191

128

Figure 9: (A) is an environment seen by the normal camera. (C)and (E) show environment (A) as seen by the normal camera
and the synchronized camera (according to the nonintrusive
projection), respectively, while (G) is being projected from the
projector. (B), (D), and (F) are Canny edge detection results of (A),(C), and (E), respectively.

Initial camera setup: V-sync synchronization: large aperture: short exposure time

: sequence of all color wheels
Color wheel analysis: mirror flips of the selected color: target color channel: specific synchronization delay: classify colors to three sets
Color classification(WHITE, BLACK, GRAY): three classified sets of colors
Finish DLP analysis (oﬄine): aperture, exposure time, target color channel,
synchronization delay, three sets of colors
Nonintrusive projection : convert all projected colors to the WHITE
set using the nearest-neighbor search(online)Figure 10: Procedural flow of the proposed nonintrusive projection
during oﬄine and online execution.

performed to ensure projecting only colors in the set whose
visibility cannot be sensed by the synchronized camera.
Section 6.1 shows in detail how to perform the first two
steps of the oﬄine processes (as written in Figure 10) whichmainly involve DLP color wheel analysis. Then, Section 6.2
explains the utilization of Section 6.1 for nonintrusive
projection; this section refers to the remaining step of
the oﬄine processes and the online processes, as shownin Figure 10. Note that a beam splitter (as described in
Section 4) has not yet been applied in these sections.
6.1. Color Wheel Analysis of a DLP Projector. In this section,
characteristics of the color wheels are analyzed for further
uses in Section 6.2. Because each DLP projector model owns
unique characteristics of the inside color wheel, color wheel
analysis has to be performed before using an unknown DLP

64
0
0

0.5

1
(ms)

1.5

2

2.5

0.8
(ms)

(b)

(c)

Figure 11: Color wheel analysis of the Acer P3250 DLP projector.
(a) is the overall color-wheel sequences as seen by the synchronizedcamera with starting exposure times ranging from 0 to 10 ms. (b) isdetailed mirror flip sequences for all 256 values in the red channel
with the starting exposure times ranging from 0 to 2.5 ms. (c) ismirror flip sequences at the selected starting exposure time.

projector model for nonintrusive projection. To understandthe overall characteristics of the color wheels without fullaccess to a DLP chip and its controller, we applied thecamera-based classification method proposed in [36]. Thissection briefly explains the color-wheel analysis steps andshows the results regarding our DLP projector model.
First, this analysis method requires a camera withan external trigger feature to synchronize it with a DLP
projector. Synchronization between the projector and thecamera is performed here by tapping the vertical sync signal(5 V, 60 Hz) from the computer to the projector and using
it to trigger the camera. In addition, the aperture size of thecamera must be increased whereas the shutter of the camera
must be set to open for a very short period (i.e., 0.55 ms inour setup) in order to sense the fast characteristics of thecolor wheel properly with no eﬀect from the ambient light.
All these settings refer to the initial camera setup step written
in Figure 10.
Second, we analyzed the overall sequences of the color
wheel inside our DLP projector by projecting single-color
images (corresponding to the colors of each available color
wheel of the projector) at maximum intensity through all
possible starting exposure times. Figure 11(a) was created byallowing the synchronized camera to sense these projectedcolors with diﬀerent starting exposure times. Then, we
analyzed detailed mirror flip sequences for all 256 values in12
the selected color channel (i.e., red, green, or blue) withina narrow starting exposure period. From Figure 11(a), we
selected the red channel, which is the first channel appearing
in the sequences; other color channels can be selected too, but
it will result in a longer synchronization delay time. Mirror
flip sequences were then obtained by projecting uniform red
images with intensity values ranging from 0 to 255, andwith the starting exposure times ranging from 0 to 2.5 ms.
Figure 11(b) was created by allowing the camera to sense
these red projections. A starting exposure time of 0.8 ms was
finally chosen because it provides the best distributed red
ramps, as shown in Figure 11(c).
Up to this point, the color wheel analysis step written in
Figure 10 is done and we are able to synchronize the camera
with the projector at the appropriate starting exposure time.
For our selected starting exposure time, the camera canonly see the red light of the DLP projector (correspondences
between projected red intensities and red intensities seen by
the camera are shown in Figures 11(b) and 11(c)).
6.2. Color Classification and Color Conversion. In [36], the256 red intensities in the selected time slot were classified
into three sets: WHITE, BLACK, and GRAY. WHITE refers
to colors whose projection fully turns mirrors inside the
DLP projector and transmits lights toward a surface (i.e.,bright red light as seen by the camera). BLACK refers
to colors whose projection does not flip the mirrors andtransmits no light toward a surface (i.e., no light seen by thecamera). GRAY refers to unreliable states between WHITE
and BLACK. By converting each projected color to its mostsimilar color in the WHITE or BLACK set, the work of [36]can imperceptibly embed the binary pattern to the normal
projection. Based on the same idea of three sets of colors,
our nonintrusive projection is performed by projecting only
the contents whose colors share the same classified set so thatthey blend across the projection area seamlessly as seen by thecamera. In this way, the camera is deceived and cannot see
the projected contents because they cannot be diﬀerentiatedfrom their background.
Nevertheless, the method of projecting only colors whose
red intensity contained in the specific set works only fora DLP projector model that does not have interdependentcolor channels in mirror flipping. Instead of depending on
the red channel and risking eﬀects from the other color
channels, we propose a DLP-model-independent classification that involves all color channels (i.e., red, green andblue) of projected images. Each color in the RGB color
space was projected onto the white surface (locating in astatic environment) and sensed by the camera. The projectedcolors were then classified into the three sets previously
discussed. These projection and classification refer to the last
step of the oﬄine processes according to Figure 10.
To use the three classified sets of colors during online
nonintrusive projection, we applied an approximate nearestneighbor search algorithm called best bin first (BBF) [37] sothat an arbitrary color is converted into our preferred set andvisibility of the projected contents (as seen by the camera)can be controlled. However, as mentioned in Section 6.1,
the exposure time of the camera is significantly decreased

EURASIP Journal on Advances in Signal Processingin this approach. Therefore, the camera cannot sense theenvironment properly unless there is light emitting from the
projector. This eﬀect can be seen in Figure 9(E) where no
ambient light is seen by the camera and the environment isonly revealed inside the projector frustum area. For, further
environment analysis purpose, this means that we need toilluminate the environment while projecting nonintrusive
contents. Hence, all projected colors must be converted to the
WHITE set to ensure both seamless visibility in the camera
image and proper environment illumination. The final resultof our nonintrusive projection is shown in Figures 9(E)and9(F). It can be seen that the Canny edge detection result issimilar to that of a conventional vision-based system where
there is no projection (as shown in Figures 9(A)and 9(B)).

7. Proof of Concept
To prove the proposed framework, we implemented a basic
program in a small-scale laboratory setting to demonstrate
the concept of real-time object augmentation in a 3-D
environment using the procedural flow as illustrated in
Figure 1. The projector and camera were colocated and thencalibrated by projecting a known pattern onto a planar whitesurface placed at a distance of 60 cm from the front edge of
the wooden base. The calibration was performed oﬄine bymanually locating 25 pairs of correspondences (δ = 25 in(2)) and computed the homography matrix as described in
Section 4.2.
In the program, 2-D Gabor filter [38], known to be usefulin segregating textural regions, is used as the base detector
and any area whose Gabor filter response magnitude is low
will be considered the target object for augmentation. In
other words, the target objects for this program are textural
regions or surfaces that have either nonuniform reflections
or discontinuities. Based on this assumption, we detectedareas of the target object inside the projector frustum anddrew bounding rectangles around each detected area usinga randomly assigned nonintrusive color. The detection is
performed here in a single-frame analysis style and the raw
detection data is utilized without further refinement. The
purpose of this program is to show that our framework
enables a visual detector to be used in the same way
as nonprojector systems and can solve the fundamental
problems of a vision- and geometry-based pro-cam system
regardless of 3-D shape of the object or a multidepth
plane. Nevertheless, despite the simple object detection, thisprogram completely neglects other vision-based algorithms
for eliminating noises, identifying an object’s shape, tracking
the target object, and so forth. Therefore, existing noises are
not eliminated and the target object may not be perfectly
extracted.
Figure 12 shows the program in action with the yellow
drawn over the camera image representing the multiscale
Gabor detection result. The snapshot images of the actual
surface were also shot from an asynchronous separate camera
in order to show how a human actually saw the projectedaugmentations. Figures 12(a) and 12(b) show results on
nonplanar surfaces whereas Figures 12(c) and 12(d) display
results on slanted and angled surfaces. Finally, Figures 12(e)EURASIP Journal on Advances in Signal Processing

13

(a)

(b)

(c)

(d)

(e)

(f)

Figure 12: Demonstration of the proposed framework using 2-D Gabor filter as the base detector. Left images are images captured by thesystem camera with the multiscale detection result drawn in yellow. Right images are snapshots of the surface taken by a separate camera.
All camera images were equally enhanced here for better visualization.

and 12(f) demonstrate results on multiple planes with diﬀerent distances to the surface. Using the proposed framework,
most blue-texture areas were detected and bounding rectangles were drawn around them regardless of their geometry
and distance to the surface (the distance reflects the amountof light as seen by the camera as illustrated in Figure 5). Colors of the object also influent the detection result. As shownin Figure 12(e) the program failed to detect the brown stitch
on the doll. This is because brown is the color that blends
across the projector light (i.e., the red light illuminating
the environment) when being seen by the camera (detailed
explanation as written in Section 6). Therefore, the program
could not distinguish it from the background in camera
images.
The same procedure can be repeated when developing
applications in a real-world scenario. Instead of the simple
visual detector used in this paper, human detector, gesturerecognition or motion analysis algorithms might be used tocarefully refine the raw data and detect a user or interpret
his actions accurately in real time. Final robustness of theapplication regarding object or human detection will dependon the selected visual detector and additional refinement
algorithms. Our framework finishes its tasks here for solving
problems of a pro-cam system, leaving problems of aconventional vision system to be solved by other appropriate
vision-based algorithms.

8. Experimental Results
In this section, we discuss experiments conducted to evaluate
the three proposed approaches. All experiments were performed using an HP Pavilion dv5 Notebook PC with an Intel
Core2 Duo CPU P8600 running at 2.40 GHz. The following
devices were used in the experiments: a TechSpec plate beam
splitter 48904-J, an Acer P3250 DLP projector (Native XGA
1024 × 768 projection resolution with standard brightness
at 2000 ANSI lumens), a Dragonfly Express camera (VGA
640 × 480 captured resolution), and an ELECOM VSPA2 VGA splitter. The camera is equipped with a Tamron
13VM308AS lens, and the projector’s focus was adjusted
manually in all experiments.

Scene-Independent Pro-Cam Geometric Calibration. To determine the accuracy of the proposed pro-cam geometriccalibration, we conducted experiments on both planar andnonplanar surfaces. The oﬄine calibration was performed
using only one sample image containing 25 calibrated points(δ = 25 according to (2)). In the experiments, camera
coordinates generated by our approach were compared withactual camera coordinates located manually. For the planarand slanted surface, five experiments were conducted with
diﬀerent distances of the projector, and each experiment
was performed using 25 tested points (the number of tested
points written here is not the δ value used in the oﬄine
calibration). The same experiment with 25 tested points
was repeated on five nonplanar surfaces which are diﬃcult
to be calibrated in real time using other noncolocating
designs. Figure 13(a) shows the experimental results on theplanar and slanted surfaces whereas Figure 13(b) shows theexperimental results on the nonplanar surfaces. Snapshotsof the nonplanar experimental surface are captured by aseparate camera and shown in Figure 13(c).
According to the experimental results shown in Figure 13(a), the proposed calibration provides the narrow
range of geometric errors in both axes. In the fourth andfifth experiments, geometric errors are higher than those of
the first three experiments. This is because the fourth andfifth experiments were conducted on a slanted surface anddepth variation of the surface slightly aﬀected accuracy of thecalibration. Detail regarding this will be explained in the next
paragraph.
According to Figure 13(b), the geometric errors for these
nonplanar surfaces are similar to those of the planar andslanted surfaces, except for the third and fourth experiments.
In the two mentioned experiments, the errors along the
X axis are higher than the others, but still less than 3%compared with the width of the captured resolution. These
increases of errors are caused by the significant depth
variation of these two experimental surfaces; the variation
makes the projected image become slightly distorted whenbeing seen by the camera. Hence, points lying on thesame straight line but diﬀerent depth planes are not ideallycollinear in the camera image and results in geometric errors

14

EURASIP Journal on Advances in Signal Processing
Nonplanar surface

Planar or slanted surface
3

6

2

5
4
3

0
−9 −8 −7 −6 −5 −4 −3 −2 −1 0
−1

1

2

3

−2
−3

ΔY (pixel)

ΔY (pixel)

1

2
1
0

−16

−14

−12

−10

−8

−6

−4

−2

−4

2

4

−3

ΔX (pixel)

Root mean square: ΔX = 4.42, ΔY = 2.15
Experiment 1
Experiment 2
Experiment 3

0

−2

−5

ΔX (pixel)−1

Root mean square: ΔX = 7.44, ΔY = 2.56

Experiment 4
Experiment 5

Experiment 1
Experiment 2
Experiment 3(a)Experiment 4
Experiment 5(b)Experiment 1

Experiment 2

Experiment 4

Experiment 5

Experiment 3(c)X

Experiment 3
Y

(d)

X

Experiment 5Y(e)

Figure 13: (a) Geometric error (camera coordinates) of the proposed pro-cam geometric calibration approach on planar and slanted
surfaces. (b) Geometric error (camera coordinates) of the proposed pro-cam geometric calibration approach on nonplanar surfaces. (c)Snapshots of the nonplanar experimental surfaces. (d, e) are the captured images of the third and fifth nonplanar experiments, respectively.

of the calibration. This is shown in Figure 13(d) where there
are two groups of tested points: 8 and 17 points with andwithout bounding circles, respectively. The first group of
points lying on the whiteboard has small geometric errors
whereas the second group of points projected onto the dollin front of the whiteboard provides larger geometric errors.
However, if the depth variation of the projection surface is

not high, as shown in Figure 13(e), 3-D shapes of the surface
barely cause distortion of the projected image as seen by thecolocated camera.
In conclusion, the proposed calibration approach provides reasonably accurate results for the purpose of ubiquitous object augmentation, particularly when dealing with asurface whose geometries are not known. The accuracy of

EURASIP Journal on Advances in Signal Processing

15

(a)

(b)

Figure 14: Images in the top row were captured by the proposed pro-cam design. Images in the bottom row result from Canny edge detection
using the proposed multiscale visual detection scheme with parallel implementation.

1400
1200

Time (ms)

1000
800
600
400
200
0
2

4

6

8

10

12

14

16

18

20

Number of scales
Sequential implementation
Parallel implementation

Figure 15: Relationship between the computational time (per one
input image) and the number of scales (N) using the sequential andparallel implementation of the proposed detection scheme.

the calibration does not fall over time but may be slightly
decreased when the projection surface has significant depth
variation.
Multiscale Visual Object Detection. For the proposed detection scheme, first, we investigated its performance using theparallel implementation in diﬀerently illuminated images.
Figure 14 shows the experimental results using Canny edge
as a representative detector. All experiments share the samesettings and no further parameter adjusting is performedin each experiment. According to Figure 14, it can be seenthat the proposed scheme is capable of detecting true edges
inside the projector frustum area in all images. Noise edges
are visible inside the detected objects given the reflection of
the projector’s light upon the object’s surface.
To determine the speed eﬃciency of the parallel implementation, Figure 15 shows the computational time usedduring the multiscale detection in the sequential and parallel

implementations versus the number of scales (N). Ourparallel implementation used the OpenMP parallel programming architecture. Therefore, the parallel capability islimited to our Core2 Duo CPU. According to Figure 15,
the times used by the sequential implementation increaselinearly while the times seem to increase exponentially inthe parallel implementation. The parallel implementation
uses less time than the sequential does when the number ofscales is less than 8. This can be explained by the conceptof parallel programming in which the ratio of the numberof data transfers to the number of executed commands
must be small; otherwise, executing in a sequential manner
on a single CPU is faster. From our experiments, thenumber of scales from four to six is adequate. Hence, using
OpenMP programming with a recent processor providing
more than two cores should allow for easy acceleration.
Another alternative is to use GPU programming, but major
reimplementation of the existing program may be required.
Nonintrusive Projection. To evaluate the nonintrusive projection, we randomly generated 64 colors and converted theminto the WHITE set using the proposed color conversion
approach. The converted image was then projected andsensed by the synchronized camera. Figure 16 shows theexperimental results including conversion of the same image
to the BLACK set for comparison. From images capturedby the camera shown in Figure 16, our classification andconversion approach can perform eﬃciently and the surface
remains well illuminated when projecting colors in the
WHITE set. However, the projection area as seen by thesystem camera is not ideally seamless when projecting colors
in the WHITE set. This is due to the range of the threshold set
during the color classification (Section 6.2). By the way, thesetraces are not clear enough to intrude on visual environment
analysis.

9. Conclusion
In this paper, we proposed a complete vision-based framework for real-time object augmentation applications using

16

EURASIP Journal on Advances in Signal Processing

Experiment 1

Experiment 2(a) Original image(b) WHITE image(c) BLACK image

Figure 16: Color conversion using the proposed approach. (a) is the randomly generated 64 colors before conversion. (b, c) are (a) images
after converted to the WHITE and BLACK set, respectively. Images in (b) represent the concept of nonintrusive projection used in this paper.
Note that for each paired image, the right image shows the environment as seen by the synchronized camera when the left image is being
projected from the projector.

a portable pro-cam device. Only the geometric aspect of thesystem is concerned in order to detect the target object andproject real-time 2-D augmentation in the desired geometry.
The framework consists of colocating the projector andcamera using a beam splitter so that scene-independent
geometric calibration can be accomplished using perspective
transformation model. Projective interference problem issolved by synchronizing the camera with the DLP projector
at a specific time slot, allowing real-time projection andreal-time visual object detection to be done simultaneously
without visually interfering with each other. Finally, theproblem of variations of lighting introduced by our proposed
design is solved using the multiscale visual detection scheme
together with parallel programming.
The proposed framework is truly portable, requires nostationary device in the workspace, and attaches no marker
to the target object or the projection surface. Additionalalgorithms can be added to allow more realistic and smoothobject augmentation in an actual application.

References[1] P. Mistry and P. Maes, “SixthSense: a wearable gesturalinterface,” in Proceedings of the SIGGRAPH ASIA, Emerging
Technologies and Sketches, 2009.
[2] R. Raskar, J. Van Baar, P. Beardsley, T. Willwacher, S. Rao, andC. Forlines, “iLamps: geometrically aware and self-configuring
projectors,” in Proceedings of the ACM SIGGRAPH, pp. 809–
818, July 2003.
[3] M. Lochtefeld, S. Gehring, J. Schoning, and A. Kruger,“ShelfTorchlight: augmenting a shelf using a camera projector
unit,” in Proceedings of the 1st Workshop on Personal Projection(UbiProjection ’10), 2010.
[4] S. Borkowski, O. Riﬀ, and J. Crowley, “Projecting rectified
images in an augmented environment,” in Proceedings ofthe IEEE International Workshop on Projector-Camera Systems(PROCAMS ’03), 2003.
[5] J. C. Lee, S. E. Hudson, J. W. Summet, and P. H. Dietz,“Moveable interactive projected displays using projector based
tracking,” in Proceedings of the Annual ACM Symposium on
User Interface Softaware and Technology (UIST ’05), pp. 63–72,
2005.

[6] K. Hosoi, V. N. Dao, and M. Sugimoto, “CoGAME: manipulating by projection,” in Proceedings of the ACM SIGGRAPH,
Emerging Technologies (SIGGRAPH ’07), 2007.
[7] M. Kanbara, A. Nagamatsu, and N. Yokoya, “Augmented
reality guide system using mobile projectors in large indoor
environment,” in Proceedings of the Workshop on Personal
Projection (UbiProjection ’10), 2010.
[8] D. Cotting and M. Gross, “Interactive environment-aware
display bubbles,” in Proceedings of the 19th Annual ACM
Symposium on User Interface Software and Technology (UIST’06), pp. 245–254, October 2006.
[9] X. Cao and R. Balakrishnan, “Interacting with dynamicallydefined information spaces using a handheld projector and apen,” in Proceedings of the 19th Annual ACM Symposium on
User Interface Software and Technology (UIST ’06), pp. 225–
234, October 2006.
[10] D. Molyneaux, H. Gellersen, G. Kortuem, and B. Schiele,“Cooperative augmentation of smart objects with projectorcamera systems,” in Proceedings of the 9th International
Conference on Ubiquitous Computing (UbiComp ’07), vol.
4717, pp. 501–518, September 2007.
[11] J. Schöning, M. Löchtefeld, M. Rohs, A. Krüger, and S. Kratz,“Map torchlight: a mobile augmented reality camera projector
unit,” in Proceedings of the 27th International Conference
Extended Abstracts on Human Factors in Computing Systems(CHI ’09), pp. 3841–3845, April 2009.
[12] M. Fiala, “Automatic projector calibration using selfidentifying patterns,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR ’05), 2005.
[13] R. Raskar, J. van Baar, and J. X. Chai, “A low-cost projector
mosaic with fast registration,” in Proceedings of the IEEE Asian
Conference on Computer Vision (ACCV ’02), 2002.
[14] R. Raskar and P. Beardsley, “A self-correcting projector,”in Proceedings of the IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, pp. II504–II508,
December 2001.
[15] J. van Baar, T. Willwacher, S. Rao, and R. Raskar, “Seamless
multi-projector display on curved screens,” in Proceedings ofthe Workshop on Virtual Environments (EGVE ’03), 2003.
[16] W. Sun, X. Yang, S. Xiao, and W. Hu, “Robust checkerboard recognition for eﬃcient nonplanar geometry registration in Projector-Camera systems,” in Proceedings of the
5th ACM/IEEE International Workshop on Projector Camera
Systems (PROCAMS ’08), August 2008.

EURASIP Journal on Advances in Signal Processing[17] Y. Ding, J. Xiao, K. H. Tan, and J. Yu, “Catadioptric projectors,”in Proceedings of the IEEE Computer Society Conference on
Computer Vision and Pattern Recognition Workshop (CVPR’09), pp. 2528–2535, June 2009.
[18] A. Kushal, J. Van Baar, R. Raskar, and P. Beardsley, “A handheld
projector supported by computer vision,” in Proceedings of the
7th Asian Conference on Computer Vision (ACCV ’06), vol.
3852, pp. 183–192, 2006.
[19] T. Johnson and H. Fuchs, “Real-time projector tracking on
complex geometry using ordinary imagery,” in Proceedingsof the International Workshop on Projector-Camera Systems(PROCAMS ’07), 2007.
[20] R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H.
Fuchs, “Oﬃce of the future: a unified approach to image-basedmodeling and spatially immersive displays,” in Proceedingsof the Annual Conference on Computer Graphics (SIGGRAPH’98), pp. 179–188, July 1998.
[21] A. Grundhöfer, M. Seeger, F. Häntsch, and O. Bimber,“Dynamic adaptation of projected imperceptible codes,” inProceedings of the 6th IEEE and ACM International Symposiumon Mixed and Augmented Reality (ISMAR ’07), November
2007.
[22] H. Park, M.-H. Lee, B.-K. Seo, Y. Jin, and J.-I. Park, “Content
adaptive embedding of complementary patterns for nonintrusive direct-projected augmented reality,” in Proceedings ofthe International Conference on Human-Computer Interaction(HCI ’07), vol. 4563, pp. 132–141, 2007.
[23] M. Waschbüsch, S. Würmlin, D. Cotting, F. Sadlo, and
M. Gross, “Scalable 3D video of dynamic scenes,” VisualComputer, vol. 21, no. 8–10, pp. 629–638, 2005.
[24] S. G. Narasimhan, S. J. Koppal, and S. Yamazaki, “Temporaldithering of illumination for fast active vision,” in Proceedingsof the 10th European Conference on Computer Vision (ECCV’08), vol. 5305, pp. 830–844, 2008.
[25] S. J. Koppal and S. G. Narasimhan, “Illustrating motion
through DLP photography,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR ’09),
pp. 9–16, 2009.
[26] Z. Wang, Z. Miao, and C. Zhang, “Extraction of highresolution face image from low-resolution and variant illumination video sequences,” in Proceedings of the 1st International
Congress on Image and Signal Processing (CISP ’08), pp. 97–
101, May 2008.
[27] J. Lou, H. Yang, W. Hu, and T. Tan, “An illumination invariantchange detection algorithm,” in Proceedings of the IEEE Asian
Conference on Computer Vision (ACCV ’02), pp. 13–18, 2002.
[28] T. Tanaka, A. Shimada, D. Arita, and R.-I. Taniguchi, “Object
detection under varying illumination based on adaptive
background modeling considering spatial locality,” in Proceedings of the 3rd Pacific Rim Symposium on Image and Video
Technology (PSIVT ’09), vol. 5414, pp. 645–656, January 2009.
[29] W. Maier, F. Bao, E. Mair, E. Steinbach, and D. Burschka,“Illumination-invariant image-based novelty detection in a
cognitive mobile robot’s environment,” in Proceedings of theIEEE International Conference on Robotics and Automation(ICRA ’10), pp. 5029–5034, 2010.
[30] P. M. Roth, S. Sternig, H. Grabner, and H. Bischof, “Classifier
grids for robust adaptive object detection,” in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition(CVPR ’09), pp. 2727–2734, 2009.
[31] K. Fujii, M. D. Grossberg, and S. K. Nayar, “A projector-camera
system with real-time photometric adaptation for dynamic
environments,” in Proceedings of the IEEE Computer Society

17
Conference on Computer Vision and Pattern Recognition (CVPR’05), pp. 814–821, June 2005.
[32] R. Sukthankar, R. G. Stockton, and M. D. Mullin, “Smarter
presentations: exploiting homography in camera-projector
systems,” in Proceedings of the 8th International Conference on
Computer Vision (ICCV ’01), pp. 247–253, July 2001.
[33] Open Source Computer Vision (OpenCV), http://opencv.willowgarage.com/wiki/.
[34] Q. Yang, K. H. Tan, and N. Ahuja, “Real-time O(1) bilateral filtering,” in Proceedings of the IEEE Computer Society Conference
on Computer Vision and Pattern Recognition Workshops (CVPR’09), pp. 557–564, June 2009.
[35] B. Schwerdtfeger, D. Pustka, A. Hofhauser, and G. Klinker,“Using laser projectors for augmented reality,” in Proceedingsof the ACM Symposium on Virtual Reality Software andTechnology (VRST ’08 ), pp. 134–137, October 2008.
[36] D. Cotting, M. Naef, M. Gross, and H. Fuchs, “Embedding
imperceptible patterns into projected images for simultaneous
acquisition and display,” in Proceedings of the 3rd IEEE andACM International Symposium on Mixed and Augmented
Reality (ISMAR ’04), pp. 100–109, 2004.
[37] J. S. Beis and D. G. Lowe, “Shape indexing using approximate
nearest-neighbour search in high-dimensional spaces,” in Proceedings of the IEEE Computer Society Conference on Computer
Vision and Pattern Recognition (CVPR ’97), pp. 1000–1006,
June 1997.
[38] S. E. Grigorescu, N. Petkov, and P. Kruizinga, “Comparison of
texture features based on Gabor filters,” IEEE Transactions on
Image Processing, vol. 11, no. 10, pp. 1160–1167, 2002.

Photographȱ©ȱTurismeȱdeȱBarcelonaȱ/ȱJ.ȱTrullàs

Preliminaryȱcallȱforȱpapers

OrganizingȱCommittee

The 2011 European Signal Processing Conference (EUSIPCOȬ2011) is thenineteenth in a series of conferences promoted by the European Association for
Signal Processing (EURASIP, www.eurasip.org). This year edition will take placein Barcelona, capital city of Catalonia (Spain), and will be jointly organized by theCentre Tecnològic de Telecomunicacions de Catalunya (CTTC) and the
Universitat Politècnica de Catalunya (UPC).
EUSIPCOȬ2011 will focus on key aspects of signal processing theory andapplicationsli ti
as listedli t d below.
b lA
Acceptancet
off submissions
b i i
will
ill beb basedb d on quality,
lit
relevance and originality. Accepted papers will be published in the EUSIPCO
proceedings and presented during the conference. Paper submissions, proposals
for tutorials and proposals for special sessions are invited in, but not limited to,the following areas of interest.

Areas of Interest• Audio and electroȬacoustics.
• Design, implementation, and applications of signal processing systems.
• Multimediald signall processing andd coding.
d• Image and multidimensional signal processing.
• Signal detection and estimation.
• Sensor array and multiȬchannel signal processing.
• Sensor fusion in networked systems.
• Signal processing for communications.
• Medical imaging and image analysis.
• NonȬstationary, nonȬlinear and nonȬGaussian signal processing.

Submissions
Procedures to submit a paper and proposals for special sessions and tutorials will
be detailed at www.eusipco2011.org. Submitted papers must be cameraȬready, no
more than 5 pages long, and conforming to the standard specified on the
EUSIPCO 2011 web site. First authors who are registered students can participate
in the best student paper competition.

ImportantȱDeadlines:
P
Proposalsȱforȱspecialȱsessionsȱ
l f
i l
i

15 D 201015ȱDecȱ2010

Proposalsȱforȱtutorials

18ȱFeb 2011

Electronicȱsubmissionȱofȱfullȱpapers

21ȱFeb 2011

Notificationȱofȱacceptance23ȱMay 2011

SubmissionȱofȱcameraȬreadyȱpapers
Webpage:ȱwww.eusipco2011.org

6ȱJun 2011

HonoraryȱChair
MiguelȱA.ȱLagunasȱ(CTTC)
GeneralȱChairAnaȱI.ȱPérezȬNeiraȱ(UPC)
GeneralȱViceȬChair
CarlesȱAntónȬHaroȱ(CTTC)
TechnicalȱProgramȱChair
XavierȱMestreȱ(CTTC)
TechnicalȱProgramȱCo
Technical
Program CoȬChairs
ChairsJavierȱHernandoȱ(UPC)
MontserratȱPardàsȱ(UPC)
PlenaryȱTalks
FerranȱMarquésȱ(UPC)
YoninaȱEldarȱ(Technion)
SpecialȱSessions
IgnacioȱSantamaríaȱ(Unversidadȱ
deȱCantabria)MatsȱBengtssonȱ(KTH)
Finances
MontserratȱNájarȱ(UPC)
Montserrat
Nájar (UPC)
Tutorials
DanielȱP.ȱPalomarȱ
(HongȱKongȱUST)BeatriceȱPesquetȬPopescuȱ(ENST)
PublicityȱStephanȱPfletschingerȱ(CTTC)
MònicaȱNavarroȱ(CTTC)
Publications
AntonioȱPascualȱ(UPC)
CarlesȱFernándezȱ(CTTC)
IIndustrialȱLiaisonȱ&ȱExhibitsdi l Li i& E hibi
AngelikiȱAlexiouȱȱ
(UniversityȱofȱPiraeus)
AlbertȱSitjàȱ(CTTC)
InternationalȱLiaisonJuȱLiuȱ(ShandongȱUniversityȬChina)
JinhongȱYuanȱ(UNSWȬAustralia)
TamasȱSziranyiȱ(SZTAKIȱȬHungary)RichȱSternȱ(CMUȬUSA)RicardoȱL.ȱdeȱQueirozȱȱ(UNBȬBrazil)