SketchSpace: Designing Interactive
Behaviors with Passive Materials
David Holman

Hrvoje Benko

Human Media Lab

Microsoft Research

Queen’s University

One Microsoft Way

Kingston, ONRedmond, Wa

holman@cs.queensu.ca

benko@microsoft.com

simplifies the process of generating an interactive
device sketch and supports rapid exploration of designsolutions.

ACM Classification Keywords
H.5.2 [Information interfaces and presentation]: UserInterfaces. – Graphics User Interfaces

Abstract
This paper presents SketchSpace, a system that allows
designers to interactively sketch [3] device’s interactive
behaviors by imbuing digital functionality to passivematerials. SketchSpace requires no augmentation of
the device itself, but instead it uses a depth-sensing
Kinect camera to simulate the use of hardware sensors
by using depth information to infer an object's threedimensional position, motion, proximity, shape,deformations, and touch events on its surface. A
designer can map these inputs to desktop applications
in real-time and thus experiment with different
interactions. We showcase how SketchSpace can beused to prototype two devices: from tilt sensitive mice
to bendable displays. In general, we discuss how thisCopyright is held by the author/owner(s).
CHI 2011, May 7–12, 2011, Vancouver, BC, Canada.
ACM 978-1-4503-0268-5/11/05.

General Terms
Design, Human Factors

Keywords
Sketching user interfaces, rapid prototyping, depth
sensing camera.

Introduction
During the design process, industrial prototypes are
critical artifacts for innovation and collaboration. Of thehundreds of prototypes a designer sculpts, few will
have interactive functionality. Bread boarding, as
designers refer to it, integrates hardware components
into a prototype to roughly approximate its final
interactive behavior. Although this approach is common
in practice, it is labor-intensive, impacts ergonomics,
and requires technical knowledge of embedded harddevices.

Figure 1. A range of materials adesigner could work with using
SketchSpace. This includes everyday
objects (wood blocks, chopsticks, spoon,baseball), deformable surfaces(Mousepad), and plastic device
prototypes (mouse).

Figure 2. A Kinect camera hovers
over the designer’s workspace. Its
depth data is used to the track thedesigner’s manipulations of a physicalprototype.

What if a designer could explore the interactive
behavior of a device prior to having a working
prototype? In this way, we envision an interactive
sketching tool that is as seamless as working with
common design materials, such as tape or foam core,
to express a physical a design. Achieving thisinteractively helps a designer experience a device
hands-on and, consequently, triggers the serendipitous
realizations arrived at with a concrete artifact [7].
With this in mind, we designed SketchSpace, alightweight environment that adds interactive behavior
to passive physical objects (see Figure 1). A designer
places an object in her workspace and works with aprototype as if it actually had physical sensors. All input
sensing in SketchSpace is achieved using Microsoft’s
Kinect’s camera (see Figure 2). The depth data allows
SketchSpace to observe a broad set of inputs, ranging
from touch events on the object’s surface, orientation,position, motion, proximity, and, among others, shapedeformations. Although these inputs are often
recognized using hardware sensors, a designer does
not have to find, attach, and work with physicalsensors. Using the depth data to infer virtual sensors
leaves the designer free to ideate, explore, and quickly
make interactive sketches.
This activity of quickly and inexpensively generating
numerous possible designs is referred to as sketching[3]. Buxton distinguishes it from prototyping anddetails how a sketch’s intentional roughness evokes
useful discussion and leads to better design solutions.
Unlike other design disciplines that operate primarily ina spatial domain, interactive sketching introduces atemporal dimension that complicates representing evensimple interactions. It is challenging —yet absolutely

necessary —for an interactive sketching tool to quickly
enumerate the design options from which to choose. To
do this, Buxton argues such a tool should enable adesigner to make quick, inexpensive, and disposablesketches. SketchSpace explores exactly this area ofsketching interactive behaviors by using only physicalmaterials and a depth camera.

Related Work
Previous work such as d.tools [5], VoodoIO [13], theCalder Toolkit [9], and BOXES [6] augment physicalprototypes with hardware sensors and provide highlevel toolkits that abstractly represent interaction to adesigner. This promotes ease of use and drasticallysimplifies the work needed to rapidly prototype
hardware interfaces. It does not, however, minimize
the physical challenge of seamlessly integrating sensors
into a prototype. The time needed to find, fasten, andfit numerous sensors is a barrier to quick anddisposable sketching [3]. Although iStuff Mobile [2]addresses this problem using a single integrated sensor
that senses acceleration, tilt, and rotation, it assumes arigid mobile form factor that cannot sense touch.
SketchSpace, instead, removes the need for mosthardware sensors and uses the depth data to generate
virtual ones. This, unlike previous examples, permits
physical prototypes to be swapped without refitting
sensors to a new device and further supports rapid
design exploration.
DisplayObjects [1], unlike physical prototyping toolkits,
relies on motion capture to rapidly prototype foam
mockups that are rendered with projected content.
Similar to SketchSpace, this minimizes the use ofphysical hardware sensors and requires only infrared
tracking markers on the mockup. Unlike SketchSpace,Figure 3. A red highlight indicates thatSketchSpace is tracking the block ofwood and its virtual sensors are active.

DisplayObjects’s motion tracking algorithms requires
the designer to build a virtual model of the mockup thatis input to the tracking engine. After this step, thedesigner is free to explore aesthetic design by dragging
projected content from a physical palette and manuallyplacing it on the mockup. Although button presses aresupported, complex input behaviors that expresschanges in a position or shape are not mapped tointeractions.

Design Concept
SketchSpace is motivated by observations of industrial
and interaction designers working at Microsoft
Hardware group. We interviewed two designers as theyworked through the design of novel computer input
devices (e.g., mice, remotes, or game controllers).
First, we observed the lead industrial designer as hesculpted different form explorations for a novelcontroller. We also interviewed the designer responsible
for the interaction techniques this controller wouldexhibit and discussed the current practice of designinginteractions.
(a)

(b)

Figure 4. A subset of the Mapping
Panel.
(a) The controls for the virtual tiltsensor. The designer enables a sensor,
sets it to a relative or absolute value,
specifies its sensitivity, and designates
the key presses to be raised when it istriggered. Similar controls for the othervirtual sensors are available.
(b) The control to add a virtual button.
The designer adjusts its position, size,and sensitivity. Up to two buttons aresupported.

Both designers confirmed that working with a functional
device is increasingly important in their work. As theinteraction designer stated: “design fails when certain
parts aren’t evaluated.” The omission of a functioning
input device increases the likelihood of failure. Although
tools like Knörig’s Fritzing [8] can make bread boarding
easier, introducing a functional prototype into thedesigner’s workflow can still be time consuming. If an
idea is too difficult to be explored or cannot berepresented, it will be avoided. Overall, the designers
wanted a sketching tool that is lightweight, easy to use,
and allows them to work together when iterating overan interactive design.

SketchSpace System
Guided by these observations, we designed
SketchSpace to allow designers to more seamlessly
sketch interactive devices. The system positions adepth sensing camera and projector above a designer’s
workspace (see Figure 2). The depth data is used to
track an object and make inferences about its input
behavior. The projector renders perspective correct
feedback to indicate that a passive object has
interactive behavior (see Figure 3). While manipulating
an object, a designer monitors its input behavior on
SketchSpace’s Mapping Panel (see Figure 4). Fromthese input values, the designer generates simple rulesthat trigger interactive actions on a Desktop
application. For example, bending the corner of scrap
paper can be quickly mapped to trigger a page back ina web browser as originally envisioned by Gallant et al.
[4]. Using SketchSpace in this way, the designer
iterates a device’s interactive potential.
Virtual Sensors
SketchSpace is capable of reporting the changes inseveral object states in a form of a virtual sensor
stream. Our selection of virtual sensors was motivatedby the most common actions that our observed
designers wanted to prototype when designing a newdevice. We have prototyped five virtual sensors thatserve as the basic elements for building interactions in
SketchSpace:
1.

Grasp. When a designer picks up an object,
SketchSpace understands the object is being
grasped and how much of it is covered.

2.

Touch. Designers can touch a single point on aflat or curved surface. We limit this gesture to3.

Figure 5. A designer adds a virtual
button to the mouse prototype and thenexplores tilt-based interactions.

Figure 6. An everyday flexible
material, such as a Mousepad, is used
to quickly explore the combination ofbending and tilting interactions. Notethat the projected color hue canchange with the tilt of the device.

single touch to simplify the complexity of
distinguishing natural grasps versus multiple
touch points.

encodings [10] introduces some overheard, it allows
the designer to control any application on the fly
without using any specialized programming interface.

Position. This sensor expresses the threedimensional position of an object as a designer
manipulates it. In general, it represents spatial
information such as position in three
dimensions, tilt around its horizontal andvertical axis, and rotation around its z-axis.

Usage Scenario

4.

Proximity. This sensor expresses the spatial
relationship between two objects and how nearor far from each other. In Figure 6, orexample, the distance between the designer’s
hand and flexible material is known. Ingeneral, this gesture enables new types ofproximity-based interaction to be explored.

5.

Deformation. A designer reforms an object’s
surface geometry dynamically. A piece ofpaper, for example, can be reshaped toexpress a convex or concave bend.

Mapping Gestures to Interface Actions
Each of these sensor streams must be mapped tospecific actions in the designer’s workflow. The Mapping
Panel provides an overview of input behavior and, foreach virtual sensor, shows its value, mapping rules,
and the corresponding action the rule triggers (seeFigure 4). We currently map all our actions to keyboard
and mouse events, which make it possible to easily
control most desktop applications. Like BOXES [6], thedesigner specifies the trigger condition and the key ormouse event that is sent to the active desktop
application. Although specifying these standardized key

To illustrate how designers work with SketchSpace, wepresent the exploration of a tilt-based mouse device
and an interactive sketch of a bendable display.
Sketching Mouse Interactions
With a non-functioning prototype of a new tilt-enabledmouse in hand, the lead industrial and interaction
designers use SketchSpace to quickly try out newinteractions. Using the Mapping Panel, the designers
map its 2D position to the desktop cursor and place avirtual button on its surface (see Figure 5). To get afeel for the tilt interactions, they decide to map themouse’s virtual tilt sensor to horizontal panning in
Microsoft’s WorldWide Telescope. As they tilt themouse, they monitor the sensor value in the Mapping
Panel. Iteratively, they tweak the sensitivity to indicate
when the mouse is sufficiently tilted. To pan left, thedesigners type “LEFT” in the mapping text field toindicate a left key press should be raised when this rule
is triggered. This process is repeated until tilting of themouse pan Microsoft’s WorldWide Telescope in every
direction.
Sketching a Bendable Display
SketchSpace also simplifies sketching interactions foremerging technologies. Gummi [12] presents abendable hardware prototype that simulates the use ofa flexible credit card sized display. To explore similarinteractions, a designer finds a flexible material andmanipulates its while monitoring the Mapping Panel’svirtual bend sensor (see Figure 6). The designer uses

Microsoft’s WorldWide Telescope and bends thematerial to zoom in and out. Using the tilt operations
from the last scenario, the designer navigates galaxies
by combining bending and tilting interactions.

Implementation

(a)

(b)
Figure 7. Skin densely reflects theinfrared light emitted by the Kinect. We
threshold the raw infrared image (a) andlocate the hand (b). Although smaller
candidates are sometimes found (redboxes), they are easily ignored.

SketchSpace’s is a modified version of LightSpace [14].
It is implemented in C# in a Windows 7 environment
and uses LightSpace’s libraries to acquire and process
depth data. Our single camera and projector setup issampled at 50 Hz and calibrated in a single 3D
coordinate system [14]. A grayscale image represents
the sampled depth data and computer vision
techniques use this image to identify and track up to
two objects at a time. Once identified, an object’s
surface geometry is used to infer input behaviors andgenerate virtual sensor streams for each object.
From Input Behaviors to Virtual Sensors
To sense an object’s input behaviors, the systemthresholds the depth image and performs connectedcomponent labeling [14]. The position of a sufficiently
large component indicates a potential object. From thecomponent’s bounding box, we extract the object’s
depth data and use this to generate a 3D surface mesh.
Using the mesh, we calculate an object’s volume, 3D
position, and infer its rotation around its local Z-axis. A
tracking engine compares these values to an index ofprevious values and generates a candidate match.
After an object is matched, we calculate its tilt using an
object’s vertical and horizontal major axis [14]. At thispoint, the object’s three-dimensional position, rotation,and tilt are known. After this, curvature is computed by
calculating the second derivative on the mesh using its

surface normals. The result is a grayscale image thatmaps degree of curvature to pixel brightness.
Touch
Sensing touch requires (1) segmenting the designer’shand (see Figure 7) and (2) knowing when a fingertip
deflects a surface. In SketchSpace’s setup, the closely
positioned Kinect floods the workspace with infraredlight. Skin is a bright reflector of infrared light and iseasily thresholded.
To sense touch, we sense a predictable pattern in thecurvature at the boundary between a fingertip andobject. After a dwell time of 500 ms, a touch event israised. To reduce false positives, we confirm thefingertip’s presence by indexing it in the segmented
hand image. Unlike Wilson’s use of depth sensing tosense touch [15], we do not require a staticbackground model of the touch surface. Relaxing thisrequirement allows us to sense touch on moving
objects. However, we currently limit our approach to asingle touch point as natural grasping gestures interferewith multi-touch sensing.

Discussion & Conclusion
While we designed SketchSpace with detailed feedback
from the interaction and industrial designers, it is still awork in progress. We plan to deploy SketchSpace to therapid prototyping shop in the Microsoft’s Hardware
group, and observe it in use when designing newdesigns. We would also like to more formally evaluate
our prototype to gather feedback from designers about
its efficacy as an interactive sketching tool.
Also, early informal feedback from designers indicatedthat they would like to render richer content on theirprototypes. At present, SketchSpace only projects
simple visual feedback onto the device to indicate
tracking and virtual buttons; however, most of thefeedback the user gets from seeing the actions
performed in the desktop application. We would like toextend our work to allow for interactive content to belocated on the device, much like the work of Nam et al.
[11] and Akaoka and Vertegaal [1]. This would allow
the designer to render images, video, and trigger
complex visual feedback, and enable deeper design
exploration of physical prototypes.
Finally, imbuing passive material with virtual sensors
has the potential to infer more abstract forms ofinteraction. Analyzing the depth data in a designer’s
studio space could reveal, for example, whether
someone has entered a room, how many people arepresent, and how many times someone has opened orclosed their laptop. Designing interactions for this levelof sensing is a rich area for future exploration.

References[5] Hartmann, B., Klemmer, S. R., Bernstein, M.,
Abdulla, L., Burr, B., Robinson-Mosher, A., and Gee, J.
2006. Reflective physical prototyping throughintegrated design, test, and analysis. In Proc. of ACM
UIST '06. 299-308.
[6] Hudson, S. and Mankoff, J. Rapid Construction ofFunctioning Physical Interfaces from Cardboard,Thumbtacks, Tin Foil and Masking Tape. In Proc. ofACM UIST’06. 289 – 298.
[7] Kirsh, D. and P. Maglio. On distinguishing epistemicfrom pragmatic action. Cognitive Science 18. pp. 51349, 1994.
[8] Knörig, A., Wettach, R., and Cohen, J. Fritzing – A
Tool for Advancing Electronic Prototyping for Designers.
In Proc. of ACM TEI’09. 351-358.
[9] Lee, J.C., Avrahami1, D., Hudson, S.E., Forlizzi, J.,
Dietz, P.H., Leigh, D. The Calder Toolkit: Wired andWireless Components for Rapidly Prototyping
Interactive Devices. In Proceedings of ACM DIS’04.
167-175[10] http://msdn.microsoft.com/enus/library/system.windows.forms.sendkeys.send.aspx.
Last accessed January 14, 2011.

[1] Akaoka, E. and Vertegaal, R. DisplayObject:Prototyping Functional Physical Interfaces on 3D
Styrofoam, Paper, or Cardboard Models. In Proc. ofACM TEI’10. 49-56.

[11] Nam, T., and Lee, W. Integrating Hardware andSoftware: Augmented Reality Based Prototyping
Method for Digital Products. In Extended Abstracts ofACM CHI’03. 956 – 957.

[2] Ballagas, R., Memon, F., Reiners, R., and Borchers,
J. iStuff Mobile: Rapidly Prototyping New Mobile Phone
Interfaces for Ubiquitous Computing. In Proc. of ACM
SIGCHI’07. 1107-1116.

[12] Schwesig, C., Poupyrev, I. and Mori, E. Gummi: A
Bendable Computer. In Proc. of ACM CHI;04. 263-270.

[3] Buxton, B. Sketching User Experiences: Getting theDesign Right and the Right Design. Morgan Kaufmann,San Francisco, CA, 2007.
[4] Gallant, D., Seniuk, A., and Vertegaal, R. Towards
More Paper-like Input: Flexible Input Devices for
Foldable Interactions Styles. In Proc. of ACM UIST’08.
283-286.

[13] Villar, N. and Gellersen, H. A Malleable Control
Structure for Softwired User Interfaces. In Extended
Abstracts. of ACM TEI’07. 49-56.
[14] Wilson, A., and Benko, H. Combining Multiple
Depth Cameras and Projectors for Interactions On,Above, and Between Surfaces. In Proc. of ACM UIST’10.
273-282.
[15] Wilson, A. Using a Depth Sensor as a TouchSensor. In Proc. of ACM ITS’10. 69-72