Pinch-the-Sky Dome:
Freehand Multi-Point Interactions with
Immersive Omni-Directional Data
Hrvoje Benko

AbstractMicrosoft Research

Pinch-the-Sky Dome is a large immersive installation
where several users can interact simultaneously withomni-directional data inside of a tilted geodesic dome.
Our system consists of an omni-directional projectorcamera unit in the center of the dome. The projector isable to project an image spanning the entire 360
degrees and a camera is used to track freehand
gestures for navigation of the content. The interactive
demos include: 1) the exploration of the astronomicaldata provided by World Wide Telescope, 2) social
networking 3D graph visualizations, 3) immersive
panoramic images, and 4) 360 degree videoconferencing. We combine speech commands withfreehand pinch gestures to provide a highly immersive
and interactive experience to several users inside thedome, with a very wide field of view for each user.

One Microsoft Way
Redmond, WA 98052 USA
benko@microsoft.com
Andrew D. Wilson
Microsoft Research
One Microsoft Way
Redmond, WA 98052 USA
benko@microsoft.com

Keywords
Freehand interaction, omni-directional interface,gestures, dome, curved displays.
Copyright is held by the author/owner(s).
CHI 2010, April 10–15, 2010, Atlanta, Georgia, USA.
ACM 978-1-60558-930-5/10/04.

2

ACM Classification Keywords
H5.2. Information interfaces and presentation (e.g.,HCI): User Interfaces – Input devices and strategies:
Graphical user interfaces.

General Terms
Design, Human Factors.

Introduction
Pinch-the-Sky Dome is a large immersive installation
where several users can interact simultaneously withomni-directional data inside of a tilted geodesic dome
(Figure 1). This experience is designed to immerse theusers in the omni-directional visualization and allow
them to manipulate and interact with data using
freehand gestures in mid-air without the need to wear
or hold tracking devices. In designing this experience,
we focused on exploring ways to allow the users tointeract with immersive content beyond arm’s reachthrough simple gestures and without on-body trackers.
We also aimed to highlight the increasing availability ofomni-directional content (e.g., panoramic imagery,space data, earth mapping data, etc.) and explore
effective ways of visualizing it within an immersive
curved display.

Figure 1. Pinch-the-Sky Dome (the entry gate is not shown inorder to capture the user inside).

First, we project astronomical imagery from World
Wide Telescope1 in our dome and allow the user toexplore the sky and the universe by simply moving
their hands above the projector (Figure 2a). As part ofthe experience, the users travel around the Solarsystem, visit the outskirts of the known universe, andobserve the incredible imagery from the Hubble Space
Telescope.
Second, the observers can be virtually transported
to several remote destinations by presenting highresolution omni-directional panoramic images; forexample, Apollo 12 lunar landing site, the lobby of theMicrosoft Research building, etc. (Figure 2b).

Dome Experience
The user enters the dome through the entry gate
which is designed to capture outside light. Inside, theuser is immersed in a 360 degree interactive
experience. Our 9ft (2.7m) dome can comfortably
accommodate up to 5 observers at any given time.
Inside, the observers have a choice of four different
visualizations.

Third, we show a live feed from a 360 degree
camera which can be used for omni-directional video
1

http://www.worldwidetelescope.org

3Implementation

conferencing scenarios with remote participants (Figure2c).

Our 9ft geodesic dome is constructed of cardboard
sheets following a 2V design2, using large paper clips tohold the cardboard sheets together. The dome rests on
a 30 degree tilted base built of standard construction
lumber. We wrapped base area under the dome withdark fabric to ensure light insulation. The cardboard
dome surrounds the projector and serves as the large
hemispherical projection surface. The various elementsof this construction can be seen in Figure 1.

Lastly, the observers can explore complex custom
made 3D graph visualizations (Figure 2d) showing thesocial network graph of one of the authors oranimations that highlight the immersive nature of thedome.

Figure 3. The projector-camera unit with a wide angle lensand the infrared illumination ring around it.

Figure 2. A collection of four different applications shown inthe dome: a) World Wide Telescope (e.g., Solar Systemvisualization), b) panoramic imagery (e.g., the Apollo 17 lunar
landing site), c) 360 degree video-conferencing application,and d) 3D visualization of a social networking graph. Note:
images are circularly distorted for dome projection.

In the middle of the dome, we placed a custommade omni-directional projector-camera unit (Figure3). This unit is based on the Magic Planet display from
Global Imagination, Inc3 which we previously2

http://www.desertdomes.com

3

http://www.globalimagination.com

4demonstrated in our Sphere project [2]. The Magic
Planet projector base uses a high-resolution DLP
projector (Projection Design F20 sx+, 1400x1050
pixels) and a custom wide-angle lens to project
imagery from the bottom of the device onto a sphericalsurface. In this project, we removed the sphericaldisplay surface of Magic Planet and simply projected
onto the entire hemisphere of the dome surface. The
quality of the projected image depends on the size ofthe dome; the brightness, contrast, and resolution of
the projector; and the amount of ambient light thatenters the dome. Our projector is capable of displaying
a circular image with diameter of 1050 pixels, orapproximately 866,000 pixels.

sensing prototypes. To provide IR light used in sensing,our setup requires a separate IR illumination source(i.e., the illumination ring around the lens).

Illumination ring
(IR LEDs)
Projector
Wide angle lensCold mirror
IR cut filter
IR pass filter

To enable freehand interactions above the projector
in mid-air, we reused the same optical axis of theprojection and we added: an infra-red (IR) sensitivecamera, an IR-pass filter for the camera, an IR-cut
filter for the projector, an IR illumination ring, and acold mirror. The physical layout of these components isillustrated in Figure 4.
Gesture-sensing is performed by an IR camera(Firefly MV camera by Point Grey Research4). Thiscamera is able to image the entire area of the projecteddisplay. To ensure that sensing is not affected bycurrently visible projected data, we perform touchsensing in the IR portion of the light spectrum, while
the projected display contains only light in the visiblespectrum. This light spectrum separation approach has
previously been demonstrated in many camera-based4

http://www.ptgrey.com

IR camera

Figure 4. Schematic of the omni-directional projector-camera
unit. The detail image shows the wide-angle lens and the IR
illumination ring around it.

User Interactions
The main contribution of this work is in enabling theuser to interact with omni-directional data in the dome
using simple freehand gestures above the projector
without special gloves or tracking devices. We
acknowledge that for many scenarios there areimportant benefits associated with using tracked
physical devices; for example, reduction of hand
movement and fatigue, availability of mode-switching
buttons, and haptic feedback. On the other hand,
tracked devices can be cumbersome, may be prone togetting lost, require batteries, and so on. Furthermore,in multi-user collaborative scenarios, the need to handoff a tracked device in order to be able to interact with5

the system can impede the flexibility and the fluidity ofinteraction.
One crucial freehand gestural interaction issue is the
problem of gesture delimiters, i.e., how can the systemknow when the movement is supposed to be aparticular gesture or action and not simply a natural
human movement through space. For surface
interactions, touch contacts provide straightforward
delimiters: when the user touches the surface they areengaged/interacting, and lift off usually signals the endof the action. However in mid-air, it is not often obvious
how to disengage from the 3D environment we live in.
This issue is similar to the classical Midas touchproblem. Therefore, gestures should be designed to
avoid accidental activation, but remain simple and easy
to perform and detect.
Since our projector-aligned camera is able to image
the entire dome, it is difficult to detect when the user isactively engaged with the system and when they aresimply watching or interacting with others in the dome.
We require a simple and reliable way to detect when
the interactions begin and end (i.e., the equivalent of amouse click in a standard user interface). We therefore
chose the pinching gesture (from [5]) as the basic unitof interaction. This can be seen by the camera as twofingers of the hand coming together and making asmall hole (Figure 5). This enabled us to literally pinch
the content and move it around to follow the hand, orintroduce two or more pinches to zoom in or out similar
to more standard multi-touch interactions available oninteractive surfaces.

Figure 5. The detection of pinching gestures above theprojector (left) in our binarized camera image (right). Redellipses mark the points where pinching was detected.

Figure 6. Using a pinching gesture to interact with theprojected content. The user is also wearing a headset
microphone.

We extended this basic functionality with speech
recognition in combination with a specific hand gesture:
the user may put two hands together (making in effect6a larger pinch or hole) and then speak a verbal
command which in turn switches visualization modes.

Conclusions and Future Work
Pinch-the-Sky Dome showcases how simple gesturalinteractions can greatly enhance the immersive
experience and how large wide-field-of-view displays
provide an immersive perspective of standard widely
available data. The inspiration for our work comes fromthe early work of Wellner [4] and Pinhanez et al. [3]where they imagined many interactive surfaces in theenvironment adapting to the users and their context.
While Pinhanez et al. [3] explored similar ideas while
researching interactions with a steerable projector, theywere unable to simultaneously project on a variety ofsurfaces in the environment, which we are able to do.
However, the limited brightness and resolution of
today’s projectors prevents us from fully realizing thisvision without an enclosed and perfectly dark room.
Ultimately, we would like to simply place ourprojector-camera setup in any room and use any
surface (walls, tables, couches, etc.) for both projection
and interaction, making the idea of on-demandubiquitous interactive surfaces a reality.

Acknowledgements
We would like to thank Jonathan Fay and the World
Wide Telescope team, and Mike Foody and Global
Imagination, Inc.

References[1]

Benko, H. (2009).
“Beyond Flat Surface
Computing: Challenges of Depth-Aware and CurvedInterfaces”. In Proceedings of ACM MultiMedia '09.
p. 935-944.

[2]

Benko, H., Wilson, A., and Balakrishnan, R. (2008)
Sphere: Multi-Touch Interactions on a Spherical
Display In Proceedings of the ACM Symposium on
User Interface Software and Technology (UIST). p.
77-86.

[3]

Pinhanez, C. S. (2001) The Everywhere Displays
Projector: A Device to Create Ubiquitous GraphicalInterfaces. In Proceedings of the International
Conference on Ubiquitous Computing (UBICOMP).
p. 315-331.

[4]

Wellner, P. (1993). Interacting with paper on theDigitalDesk. Communications of the ACM. 36, 7(Jul. 1993). p. 87–96.

[5]Wilson, A. (2006) Robust Computer Vision-Based
Detection of Pinching for One and Two-Handed
Gesture Input. In Proceedings of the ACM
Symposium on User Interface Software andTechnology (UIST). p. 255-258.