Proceedings of Virtual Reality International Conference (VRIC 2011), 6-8 April 2011, Laval, France.
RICHIR Simon, SHIRAI Akihiko Editors. International conference organized by Laval Virtual.

Invoked computing
Spatial audio and video AR invoked through miming

ZERROUG, Alexis, CASSINELLI, Alvaro, ISHIKAWA, Masatoshi
Ishikawa Komuro Laboratory, the University of Tokyoazerroug@gmail.com

Figure 1. “Invoked Computing” at home; mail on a pizza box (right) and virtual birds on the wall (left).

Abstract— Direct interaction with everyday objects
augmented with artificial affordances is clearly a very
efficient approach leveraging natural human interaction
capabilities. Hence the idea of conceiving ubiquitous
computing as an invisible world which can be “condensed”
on real objects. Rich Gold’s [1] described ubiquitous
computing field as an “enchanted village” in which people
discover hidden affordances in everyday objects. With thisproject, we explore the reverse scenario: a ubiquitous
intelligence capable of discovering affordances suggested orrepresented symbolically by human beings (as actions andscenarios involving objects and drawings). An example:
taking a banana and bringing it closer to the ear. The
gesture is clear enough: directional microphones andparametric speakers hidden in the room would make thebanana function as a real handset on the spot.

(graphically for instance). The system will try torecognize the suggested affordance, and actually
instantiate the represented object and its function, throughcommon AR techniques. For instance, to invoke a laptop
computer, the user could take a pizza box, open it and“tape” on its surface; to invoke a sound system, the user
could perhaps draw a vinyl, and do as if he were putting
earphones.

Keywords - Augmented Reality (AR); UbiquitousComputing; New Human Interfaces and Displays;affordances [2] detection;

II. RELATED WORKS
A calibrated camera/projector pair is a common set-up
for spatial augmented reality. On a previous work, one of
the authors used such configuration to develop a “VolumeSlicing Display” [3], a device capable of projecting a sliceof a 3d volume on a passive screen. The area wassomehow limited; works such as [4] introduce a way toincrease the interaction area by using a steerable base forthe camera/projector pair. This is the approach taken in
the present project. In addition, we propose to improve thepossibilities of interaction by using a parametric speaker

I.
INTRODUCTION
The aim of the “invoked computing” project is todevelop a multi-modal system able to turn everyday
objects into computer interfaces or communication
devices on the spot. To “invoke” an application, the user
has just to mimic a specific scenario or represent a tool

We are interested here on developing a multi-modal
AR system able to augment objects with video as well assound using this interaction paradigm. Rather than a
Human-to-Machine Interface, this could be called a
Machine-to-Human Interface.

Laval Virtual VRIC 2011 Proceedings - RICHIR Simon, SHIRAI Akihiko Editors

ZERROUG, Alexis & al., Invoked computing: spatial audio and video AR invoked through miming.
VRIC 2011 proceedings

giving the impression that sound comes from theaugmented object itself.

for the projection of the virtual display onto asurface.)

The discovery of affordances by an artificial
intelligence has been studied in the field of robotic inorder to give to machine the ability to recognize objects aswell as their specific uses. Interestingly, in [5], the authors
describe a specific learning process relying on imitation.
III. TECHNICAL DETAILSAs explained above, a particularity of the proposed
AR system is the inclusion of parametric speakers, suchas the AudioSpotlight; this technology is capable ofcreating audible sound from an array of ultrasoundsources. Due to the high directivity and low attenuation ofultrasound, the sound can be heard from very far inside aspecific area. Outside of this area, the sound is almost
inexistent. For short distance applications, the soundbounces on obstacles and create an impressive effect ofspatialized sound. By analogy to light, usual speakers
would be incandescent bulbs when parametric speakers
are flashlights. The combination of a projector and aparametric speaker on a steerable base create a reallystrong association of the sound with the image andincrease the effect of spatialization even with a single
source.

•In the case of marker-less interaction, we areplanning to use a time-of-flight camera (e.g.
SwissRanger or Kinect) to detect flat surfaces
for projection or other pattern for tracking.

•Edge detection on the background might also
be used to detect possible projection surfaces
on the wall.

IV. DEMONSTRATION
We would like to create a living room setup with asofa, a table and everyday objects that could be used asinterfaces. If possible, we prefer to have white walls inorder to use them as projection surfaces. Our system has
to be fixed to the ceiling, so we need a support on the topof the booth. We also need a quiet sound environment tobe able to hear the sound from the parametric speakers

Figure 3. Desired set-up for Laval Virtual Revolution 2011

REFERENCES

Figure 2. System detail

The most challenging part of this proposal is theautomatic detection of suggested affordances. Object
discovery and classification using perceived affordances
have been studied in previous research [6]. In our proof ofprinciple experiment, we have implemented a system witha very reduced capacity: the system is capable of detecting
when something resembling a flat box is opened or closed,- and then infer that it may be a laptop computer. We arenow exploring several tracking techniques both fordetecting interesting objects in the scene as well as
augmenting them with light and sound:••High-speed vision systems [7] give a goodstability for the tracking of moving objects.
Marker based tracking such as ARToolkit(much appreciated for its stability and theprecision of the 3d pose estimation required

[1]
[2]
[3]

[4]

[5]

[6]

[7]

R. Gold, “This is not a pipe.” Commun. ACM 36, July 1993.
J.J. Gibson, “The theory of affordance. In: Perceiving, Acting, andKnowing”, Lawrence Erlbaum Associates, Hillsdale, NJ, 1977.
A. Cassinelli and M. Ishikawa, “Volume Slicing Display”,
SIGGRAPH ASIA 2009, Emerging Technologies, Yokohama(2009). Emerging Technologies Catalog, p.88.
C. Pinhanez, “Using a steerable projector and a camera totransform surfaces into interactive displays”, In CHI '01 extended
abstracts on Human factors in computing systems (CHI '01).
ACM, New York, NY, USA, 2001.
L. Montesano, M. Lopes, A. Bernardino, J. Santos-Victor,"Learning
ObjectAffordances:From
Sensory--Motor
Coordination to Imitation," Robotics, IEEE Transactions on ,vol.24, no.1, pp.15-26, Feb. 2008.
M. Stark, P. Lies, M. Zillich, J. Wyatt and B. Schiele, “Functional
Object Class Detection Based on Learned Affordance Cues”,
Lecture Notes in Computer Science, Volume 5008/2008, 2008.
T. Komuro, T. Tabata, M. Ishikawa, “A Reconfigurable
Embedded System for 1000 f/s Real-Time Vision”, IEEE
Transactions on Circuits and Systems for Video Technology, Vol.
20, No. 4, pp. 496-504, 2010.

Laval Virtual VRIC 2011 Proceedings - RICHIR Simon, SHIRAI Akihiko Editors