© The Author 2013. Published by Oxford University Press on behalf of The British Computer Society. All rights reserved.
For Permissions, please email: journals.permissions@oup.com
doi:10.1093/iwc/iws018

The Design of Organic User Interfaces:
Shape, Sketching and Hypercontext
David Holman1,∗ , Audrey Girouard2 , Hrvoje Benko3 and Roel Vertegaal1
1 School of Computing, Queen’s University, 557 Goodwin Hall, Kingston, Ont., Canada K7L 3N6
2 School of Information Technology, Carleton University, Azrieli Pavilion, Room 230L, 1125 Colonel By

Drive, Ottawa, Ont., Canada K1S 5B6
3 Natural Interaction Research – Microsoft Research, Bldg. 99, One Microsoft Way, Redmond WA

98052-6399, USA∗Corresponding author: holman@cs.queensu.ca

STUDY HIGHLIGHTS• The art of user interface design is on the cusp of a revolutionary change.
• Flexible display materials dramatically alter how computer interfaces can be designed.
• Technology is moving towards that of a computational material.
Keywords: organic user interfaces; computational material; non-planar interface
Editorial Board Member: Kasper Hornbæk
Received 3 October 2011; Revised 1 June 2012; Accepted 1 October 2012

1.

INTRODUCTION

The art of user interface design is on the cusp of a revolutionarychange; one that will require designers to think about theeffect of a material and a form on a design. Traditionally,
today’s computer displays afford interactions that take place
on a flat surface. With the arrival of sensing touch on threedimensional surfaces (Benko et al., 2008; Han, 2005; Rekimoto,2002; Wilson, 2010) and the advancement of flexible display
technologies1 (Lahey et al., 2011; Schwesig et al., 2004), a
new category of non-planar interfaces have emerged. This nextgeneration interface design might have curved and deformable
1 Specifically, Flexible Organic Light Emitting Diodes (FOLEDs) andFlexible Electrophoretic Ink (E Ink) (Harrison et al., 2011).

interface forms (Holman and Vertegaal, 2008), and may even
include complex shapes that transform themselves. Snaplet, anearly example, is a paper-like computer that can change its flat
shape to contour to the curve of a user’s arm when worn (Tarun
et al., 2011). The design of these three-dimensional interfaces
present new usability challenges to designers as they explore
the nuances that shapes pose to interaction.
What impact will this have on user-centered design then? This
new consideration for both interaction and form suggests thatinterfaces have a new property of a three-dimensional interface
shape (one that—unlike before—encourages designers tocontextualize interaction in physical shape). Unlike the flat
surfaces of tablets, mobile phones or desktop computers, thisinteractive form will be wrapped around everyday objects

Interacting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

With the emergence of flexible display technologies, it will be necessary for interface designers tomove beyond flat interfaces and to contextualize interaction in an object’s physical shape. Groundedin early explorations of organic user interfaces (OUIs), this paper examines the evolving relationshipbetween industrial and interaction designs and examines how not only what we design is changing,but how we design too. First, we discuss how (and why) to better support the design of OUIs: how
supporting sketching, a fundamental activity of many design fields, is increasingly critical and why a‘hypercontextualized’ approach to their design can reduce the drawbacks met when everyday objects
become interactive. Finally, underlying both these points is the maturation of technology to that ofa computational material; when interactive hardware is seamlessly melded into an object’s shape,the ‘computer’ disappears and is better seen as a basic design material that, incidentally, happens tohave interactive behavior.

134David Holman et al.
ceramics used by an industrial designer. This transformation,of course, injects a three-dimensional form in the user interface
design, one that affords an abundance of shapes, curves andgeometries.
This, of course, raises fundamental questions for the userinterface design: how should this organic form be designed?
What type of tools will designers need to ideate over thedesign of these interactive shapes and objects? With thepotential for interactive form to be integrated seamlessly into
an everyday object, what overarching design objectives areimportant to mitigate a world filled with an abundance oftechnological distractions? How will designers rapidly sketch
using computational materials? These are only a selection of
the questions encountered in this new design space.
As a starting point, we position that not only do organic userinterfaces (OUIs) change what we design, but they have a dramatic impact on how we design too. First, designers of OUIs
need repurposed sensors, ones that are more like computational
materials and allow them to more seamlessly sketch interactive
form (Buxton, 2007). As interactive shapes advance to higher
spatiotemporal complexities, from Gummi’s bendable credit
card [00] to Benko et al.’s (2008) Sphere, there will be anincreasing need to rapidly try out—or sketch ideas without
investing weeks in creating a fully functioning hardware prototype. We discuss how this activity is increasingly relevant in an
early sketching tool called SketchSpace, a system that removes
the need for hardware input and allows passive, non-functioning
prototypes to have interactive behaviors. We discuss how it canbe used for rough ideation and contrast it against the limitations
encountered when building PaperPhone’s (Lahey et al., 2011)
more elaborate hardware prototype.
Finally, what does it mean when designers have thecomputational materials to rapidly ideate and place interaction
anywhere, at any point on an object’s surface? The potential forinteraction everywhere forces us to consider a highly interactive
world. We argue that to make the interactive form more
purposeful, designers should contextualize interaction in an
object’s shape and limit functionality to one or two behaviors at
most. This minimalist hypercontextualized approach cautiously
restricts interactive functionality, one that is increasingly
relevant for the design of OUIs.

2.

RELATED WORK

An organic design is an emerging research area, one thatcertainly overlaps with the aspects of Tangible (Ishii and Ullmer,
1997) and Natural User Interface (NUI) Design (Wigdor andWixon, 2011), even Ubiquitous Computing (Weiser, 1991). It
is important to understand where these boundaries are mostpronounced. We start from the definition of an OUI (Holman
and Vertegaal, 2008):
a computer interface that uses a non-planar display as a primarymeans of output, as well as input. When flexible, OUIs have theInteracting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

(Vertegaal et al., 2011). The morning newspaper, credit cards,
light switches or even kitchen plates would maintain their
original identity, yet could be augmented with a seamlessand interactive high-resolution display skin. Consequently, theresult is a tighter coupling between interaction and industrial
design, one that requires new tools to better ideate and explore
this organic design space.
With early work in OUI as an overarching theme (Holman andVertegaal, 2008), this paper summarizes previous explorationsin this new space of non-planar interface design and focuses on
the challenges met when introducing ‘shape’ into user-centered
design. Flexible displays, and the physical expressivenessthey afford, suggest a higher dimensionality to an interface
design, one where, similar to an industrial design, shape hasa critical role. In this context, we argue that this flexible display
material and the potential for interactive form will have a
profound impact on the design process. The interaction designertrained for a traditional—meaning flat—software design may
not be appropriately equipped for this new design context.
An interactive form, distinctively, needs to be designed (orsculpted even) with an awareness of three-dimensional form
and its interplay with a material. As a web design was graduallytaken over by trained graphics designers, in a similar trend,industrial designers will be the first to pragmatically encounterand reflect on everyday objects that have interactive behaviors.
This is not to say that an interaction design will be antiquated.
Instead, a new profession will arise, one that blends industrial
and interactive perspectives and examines design scenarios
previously impossible.
Our earlier work in OUIs both suggests (Holman andVertegaal, 2008) and details this merger of industrial andinteraction designs (Holman and Vertegaal, 2011a, 2011b;
Vertegaal, 2011a) in a new design field. Initial user evaluations
noted how the hands-on form of apaper windows (Holman et al.,2005) or interactive pop can (Akaoka and Vertegaal, 2010) felt
not like a computer at all, but more like a ‘real’thing encounteredin everyday life. Grabbing a paper window or bending
PaperPhone (Lahey et al., 2011) to find a contact, or zooming
into a map on the sphere is nothing like using a computer. Later
explorations led to the development of new and more expressive
sensors like Tactiletape (Holman and Vertegaal, 2011a, 2011b),
a touch-sensitive tape that is curved, bent and easily wrappedaround a non-planar prototype. Not only were interfaces curved
or flexible, but the tools used designed them were too.
What exactly do these developments mean? We argue
that these examples point to technology’s maturation to thatof a computational material. When interactive hardware iscommoditized and seamlessly melded with everyday things,
such that an interactive display skin can be shrink-wrappedover its entire surface, the idea of a ‘computer’ disappears andis better described as a basic design material that, incidentally,
happens to have an interactive behavior. Computation, in thisnew context, should be seen less like hardware and moreas a basic design material, not unlike the plastics, wood orDesign of OUIs: Shape, Sketching and Hypercontext
ability to become the data on display through deformation, either
via manipulation or actuation.

Examples of Organic Design

To place the transition from traditional to organic designin context, we discuss enabling a sensing technology andevaluation of a non-planar form. For simplicity, we categorize
each work as either rigid or deformable.
2.1.1. Interaction with rigid organics
Rigid curved touch devices vary considerably in shape, size andsensing technology. Rekimoto’s SmartSkin exemplifies the useof capacitance sensing to enable multi-finger sensing acrossan array of surfaces (Rekimoto, 2002). Benko et al.’s (2008)
Sphere affords users a large spherical display that diffuses
illumination to track multi-touch gestures along its surface.
Song et al.’s MTPen prototype wraps a capacitance multi-touch
sensor around a pen to detect gripping gestures during drawing
tasks (Song et al., 2011). The UnMousePad (Rosenberg andPerlin, 2009) implements a force-resistive technique to produce
a thin flexible pad that senses multi-touch input. Wimmer et al.’suses time domain reflectometry (TDR; Wimmer and Baudisch,2011) to register multiple touch points as ‘faults’ in metallic
tape, a material that can be adhered to non-planar shapes.
2.1.2. Evaluation with rigid organics
Roudaut et al. (2009) were the first to present an in-depth
evaluation of curvature and single-touch target accuracy on
spherical rigid surfaces. Across nine levels of curvature, theyplaced multiple targets on protruding and intruding surfaces.
Subjects drag their finger along a series of handcrafted spherical
FTIR surfaces (Han, 2005) until they reached a target. Theirresults indicate that convexity increases the pointing accuracy
and that concavity produces larger errors offsets between thetarget and finger and that when acquiring targets on a downward
slope, subjects tended to hook their finger posture. These results
introduce two recommendations to designers. First, knowledgeof how curvature and slope impact target accuracy suggest
how engineers can introduce corrective offset in device drivers.
Secondly, placing targets, like buttons or other widgets, onpoints of extreme convex curvatures make them easier to target.
Weiss et al. also evaluate dragging gestures on BendDesk(Weiss et al., 2010), a multi-touch surface modeled on the
curved desk in Sun’s Starfire video. They found that dragging,when compared with dragging across the connecting curve, isfaster and straighter on the flat surface.
Benko et al.’s (2008) sphere is a quintessential example of thechallenges that may arise when moving from a flat to non-planar
design. Their qualitative findings propose a number of design
considerations unique to spherical surfaces, including how tobest place content in relation to the user’s head position and,more generally, that the variability of the spherical geometry
leads to an interaction language noticeably different from flat.
2.1.3. Interaction with deformable organics
Ishii’s Illuminating Clay (Piper et al., 2002) is an earlytangible deformable interface that uses clay as an input materialInteracting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

Its foundation rests on the development of thin-film flexible
displays, the ability to caste them in non-planar shapes, andthe potential to sense touch and render a dynamic content at
any point along the surface. Unlike ubiquitous computing’s
vision of pad, tabs and boards scattered everywhere (Weiser,
1991), OUIs assume no fixed form factor and instead see
the potential to alter shape as something essential to design.
Although more recent ubiquitous examples use projection toplace pixels on everyday objects, such as Harrison’s shoulder
mounted OmniTouch (Harrison et al., 2011), OUI is firmly
motivated by the use of actual flexible displays, one tightly
coupled with the industrial design of an object.
Ishii’s Illuminating Clay (Piper et al., 2002) is an earlyexample of an interface with facets of OUI; it was an interactive display made of deformable clay that projected a digital
map over its topography, blurring the boundary between input
and output (or atoms and bits). A similar example is found in
Ping Pong Plus (Ishii and Ullmer, 1997), although it lacked theability to render dynamic content on the game’s ball, a rigid
non-planar object central to the interaction. As a consequence,tangible content tended towards embodying digital informationin shape and using it as a means for input. Ishii’s subsequent
Radical Atoms (Ishii et al., 2012) transforms this delineationof input and output by assuming a hypothetical generation ofmaterials that alter shape and appearance dynamically, and as
he says, ‘so they are as reconfigurable as pixels on a screen’(Ishii et al., 2012). This focus on programmable dynamic
materiality is a distinction from the organic design. Although
OUIs may actuate or make structural deformations, they do soin the context of an object’s overall shape, one that is wrappedwith an interactive display. Radical Atoms consider shape asa volume, a mass of particles to be programmed to fluidly represent digital information. This hypothetical scenario moves
beyond the industrial design and unifies input and output at aparticle level. It positions atoms as programmable bits, an earlyexample found in ZeroN, the computer-controlled levitating
handheld sphere (Ishii et al., 2012).
NUIs, on the other hand, are formed by an approach that
focuses on designing an interface that makes a user feel andact natural (Wigdor and Wixon, 2011). It does so in a structural
framework for natural design, one that initially determines amarket’s niche and use case goals, and, among other steps, iterates until a product ships. Unlike OUI, NUI is not directly rooted
in one direct technological development: that is, the advancement in flexible display technology on which OUI is grounded.
Instead, it is more of a development process that leverages varied input devices, from the Microsoft Surface to the Kinect, toachieve its philosophical aim of a natural user experience. An
organic interface could certainly act as a means to achieve a natural design. However, an organic design—in and of itself—ismore about the design of interactive three-dimensional form.

2.1.

135

136

David Holman et al.

2.1.4. Evaluation of deformable organics
Empirically, Dijkstra et al. (2011) explore the effects ofstructural holds and flexible display rigidity on a direct input
pointing and dragging tasks. From this study, they introduce
concepts for grip, rigid and flexible zones, and conclude thatthe distribution of these zones affect the Fitt’s Law index ofdifficulty of touch pointing and dragging tasks.

3. A NEW TYPE OF DESIGN
We begin this section with an observation that the current
methodologies of user interface design, much like the charactersin Abbot’s Flatland (Abbot, 1884), are inherently limited bytheir flat dimensionality. After having built, researched andevaluated a broad set of objects with both simulated andfunctioning curved interactive displays skins, from a dynamic
pop-can (see Figure 1) (Akaoka and Vertegaal, 2010), to a paper
window (Holman et al., 2005), to a paper phone (Lahey et al.,2011), it is evident that these non-planar forms lead to interface
characteristics that are contingent on their unique shape. In

Figure 1. The Dynacan is a pop-can computer that embodies acylindrical interactive display and demonstrates an application in massconsumerism.

earlier work, we defined the three design principles of OUIs
(Input equals output, Function equals form, Form follows flow).
These serve as an initial indication of this ‘non-planar’ organic
design space (we refer the reader to Holman and Vertegaal(2008) for a detailed discussion).
3.1.

Rapidly sketching and exploring organic design

Although the design principles of OUI offers a departure
point to reason about what an organic design means, they
do not elaborate on how the methodology and tools designer
use should change to support building, ideating and rapidly
exploring an organic design. Any early examples of a nonplanar design used construction methods that were dissimilarfrom traditional software prototype, suggesting that the varied
approaches were needed when exploring a non-planar design.
Gummi used a small rigid LCD mounted on a bend-sensing
substrate to simulate a flexible credit card (Schwesig et al.,2004). Ishii used a top-down projection and computer vision tosimulate an embodied and dynamic terrain (Ishii and Ullmer,
1997). PaperWindows (Holman et al., 2005) is an earlyexample of an OUI that investigates non-planar interaction,
simulating the interactive potential of digital paper displays(see Figure 2). Using a combination of motion tracking andprojection, computer windows are rendered onto a piece of
paper giving the illusion that the paper is, in fact, an interactive
display. This metaphor is later instantiated in PaperPhone(Lahey et al., 2011), a paper computer that uses flexible E Ink
to bend the interface and express interaction (see Figure 3).
Both these interfaces extrapolate to a scenario explored in
DisplayStacks (Girouard et al., 2012), where piles of paper
computers surround us, thin enough to be tossed around, stacked
and manipulated much like real paper.
Although these non-planar interfaces explore aspects oforganic design, their ideation and iteration inevitably hinged

Figure 2. PaperWindows explores a set of interaction techniques for
digital paper interfaces.

Interacting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

for real-time manipulation of 3D terrain. Schwesig’s Gummi(Schwesig et al., 2004) envisions bendable interactions throughmanipulation of a credit card-sized flexible display. Using
projection and motion capturing, PaperWindows (Holman et al.,2005) prototypes interaction techniques for bendable paper–computer interfaces. PhotoelasticTouch (Sato et al., 2009) uses
computer-vision techniques and the photoelastic propertiesof the transparent rubber to recognize the deformations andmanipulations of elastic shapes. Vogt et al. (2004) present apuck-size malleable input surface that uses computer vision to
map the translations of a dot pattern and compute its surface
geometry in three dimensions. Cassinelli et al.’s (2005) Khronos
projector allows a user to spatially manipulate a video content
on a projected deformable display. Stevenson et al. (2011)
present an inflatable hemispherical multi-touch display thatvaries between flat, concave and convex spherical form.

Design of OUIs: Shape, Sketching and Hypercontext

137

Although platforms like iStuff Mobile (Ballagas et al., 2007)
address this problem using a single integrated sensor thatsenses acceleration, tilt and rotation, it does so by assuming
a rigid mobile form factor. Generally, a designer is limited tothe physical expressivity and sensing potential of the sensor
supported on each platform. Stated another way, the OUIs weconstruct are limited by the expressiveness of the sensors used
to create the early prototypes that are used to explore a design.
For example, in place of a custom flexible circuit, it would beeasy to attach a bend sensor to a piece of paper to explore bendinteractions. However, to support an organic design, the same
platform would also need to track the piece of paper and project
meaningful feedback on the prototype. If a designer wished to
explore richer undulating deformation-based interactions, thenumber of hardware sensors only increases.

on a fully functioning prototype (one that required a significant
investment in both time and materials to construct). To support
designers better, we must update our design tools to answer
the challenge of seamlessly prototyping interactive organic
systems. PaperPhone’s flexible backplane, on which its bend
sensors were soldered, required custom-etched circuitry that
took, at a minimum, 2–3 days to construct. In early iterations,the thickness of the copper sheet and layout of the circuit
patterned negatively affected the ergonomic of a user’s hand
position, requiring lengthy iterations. Unifying the challenges ofindustrial and interface design this way, in the form of an flexible
circuit backplane, typifies the basic challenge of working
with early organic design: shape adds a layer of complexity
to exploring interaction design, one that makes designing
for and sensing interactions on even simple interfaces, like
PaperPhone’s bending interaction, incredibly difficult. Similararguments can be made for Gummi, Illuminating Clay or any
other fully functioning organic hardware device.
3.1.1. Hardware rapid prototyping
There are certainly tools for rapid prototyping that consider
this problem of minimizing the labor involved in building
functioning hardware devices, ones that are even effective for
exploring aspects of non-planar systems. Previous hardware
prototype tools such as d.tools (Hartmann et al., 2006), VoodIO(Villar and Gellersen, 2007), the Calder Toolkit (Lee et al.,2004), BOXES (Abbot, 1884) augment physical prototypes
with hardware sensors and provide high-level toolkits that
abstract the technical complexity of hardware sensing for aninteraction designer. This promotes ease of use and drasticallysimplifies the work needed to rapidly prototype hardwareinterfaces. It does not, however, minimize the physical challengeof seamlessly integrating and affixing sensors into a prototype.
The time needed to find, fasten and fit numerous sensors isa barrier to quick and disposable sketching (Buxton, 2007).

3.1.2. Vision-based rapid prototyping
Computer vision-based platforms, like DisplayObjects (Akaoka
and Vertegaal, 2010), are one way to address this inherent
limitation of hardware toolkits. By relying on motion capture,a designer prototypes with foam mockups that are renderedwith projected content. This minimizes the use of physicalhardware sensors and requires only infrared tracking markers
on the mockup. A designer, though, has to build a virtual modelof the mockup and provide it to the motion-tracking engine in
an initial calibration step. After this step, the designer is free
to explore its design aesthetic by dragging projected content
from a physical palette and manually placing it on the mockup.
Although button presses are supported, complex input behaviors
that express changes in a position or shape are not mapped tointeractions. Although practical once properly setup, the upfront
time to instrument and calibrate each mockup is prohibitive.
3.2.

Supporting organic design

Although each of these hardware and vision-based prototypes
have advantages, they both have the same limitation: they
assume that a designer has the necessary technical aptitude tooperate these platforms. Platforms like d.tools that reduce thisdemand do so by abstracting the technical layer in a custom
software workflow, one that a designer must adopt. Whenconsidering computation as material, though, this requirement
seems prohibitive; software sketching tools should be justas expressive as tape, foam core or any other material usedby a designer when ideating an industrial object. When thecomplexity of managing and mapping the input to interaction
becomes too high, hardware returns to the forefront again;interaction is no longer just an incidental property of a materiala designer works with.
If industrial designers are the first to encounter computational
materials and the exploration of organic design, this puts
different design constraints on the usability of these prototyping
systems. We argue that to better support an organic design,
we should adopt a design behavior common to all other fields

Interacting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

Figure 3. PaperPhone is an interactive flexible paper computer thatuses bending as a way of navigating.

138

David Holman et al.

of design: sketching. Doing so would mean that building andexploring interactive designs should be as seamless as working
with other common design materials, such as tape, foam coreor others. This is exactly why computation should have similarmaterial properties.
In Sketching User Experiences, Buxton (2007) argues thatthe activity of sketching—in itself—is not about the material
form it embodies, such as an architect’s pencil drawing or an
automotive designer’s clay sculpture, but more of an abstract
activity that impacts design thinking and learning. Beyond
Wizard of Oz or Smoke and Mirror techniques, there are few
example systems that adequately support interactive sketching,or even ones that scale to the complexities posed when
considering shape and interaction in an organic design.

SUPPORTING INTERACTIVE SKETCHING

To support interactive sketching through more expressive
computational materials, our first approach was to integrate thesensing technology directly into the design materials alreadyused by designers. One example we constructed is Tactiletape(see Figure 4), a one-dimensional pliable touch sensor that looks
and behaves like a regular tape (Holman and Vertegaal, 2011a,
2011b). Tactiletape can be built from everyday supplies: an H2
pencil (resistive surface), tin foil (conductive surface) and a shelfliner (spacing material). We position it as a readily available
material in the design studio. When a designer wishes to add
touch sensitivity to an industrial prototype, they grab a roll of
Tactiletape, cut-off a piece and attach it to the surface. To sensea touch input, the designer needs only to connect two wires: one
to each of the resistive and conductive surfaces. As a hardware
sensor, its electrical behavior is similar to common sensors used

4.1.

Figure 4. A typical Tactiletape scenario. A piece is cut from the roll(foreground) and attached to an hourglass. An Arduino relays touch
data to a desktop computer.

SketchSpace

These design questions and observations led to the design andimplementation of SketchSpace (Holman and Benko, 2011),
a lightweight environment that adds implicit input sensing topassive physical material (see Figure 5). We envision a designer
tasked with exploring an interactive water bottle, one example
of an organic device, using a system like SketchSpace. The
designer would rummage the design studio looking for a waterbottle. After finding one and placing it in their workspace, theywould touch, swipe, shake, squeeze and repeatedly interact withit using the digital tabletop interface to explore different digital
mockups of its interface (see Figure 5). In doing so, the designer
quickly and rapidly explores multiple interactive design paths,
before settling on those that feel right.
SketchSpace is a tool and set of interaction techniques thatallows a designer to use passive materials as a means to roughlyInteracting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

4.

with Arduino. Using Tactiletape, the designer can explore thetouch input on a variety of curved and deformable surfaces:
spheres, coffee cups, bracelets, paper, credit cards, dynacans(Akaoka and Vertegaal, 2010) and so on.
Wimmer et al.’s TDR (Wimmer and Baudisch, 2011)is another example of a computational material and is analternative to Tactiletape’s linearly resistive sensing. It canbe used to quickly make a non-planar prototype multi-touchsensitive, using a continuous copper strip. When a signal ispulsed along this strip, a finger’s presence generates a reflective
echo, one that can be interpreted as touch input (similar to theway breaks are detected in cross continental undersea cables).
At present, TDR requires an oscilloscope to process the analog
echo signal, increasing its material complexity.
In either case, both Tactiletape and TDR require adesigner to have some technical knowledge before exploringan organic design. Although we envision computational
materials occupying an essential role in high-fidelity prototypes
eventually finished products, what if a designer could explore
the interactive behavior of a device without a functioning
prototype or computational material? That is, what if we could
remove the need for hardware layer or computational material
from the early design and ideation stage altogether?
With this question in mind, we started a dialogue withindustrial designers at Microsoft’s Hardware Input Group(Holman and Benko, 2011) to see how they approach the
exploration of interactive systems. Our observations confirmedthat working with an interactive functional prototype isincreasingly important for an industrial design and that, ingeneral, the number of hardware components required forearly prototypes is on the rise. Although tools like Knörig’s
Fritzing (Knörig et al., 2009) can make integrating circuitry
and building custom input devices less ‘technical’, introducing
a functional prototype into the designer’s workflow can still
be time consuming, even more so for the complex surface
geometries encountered in an organic design.

Design of OUIs: Shape, Sketching and Hypercontext

139

Figure 5. A Kinect camera hovers over the designer’s workspace. Its
depth data are used to track the designer’s manipulations of a physicalprototype. A Pico projector renders interactive content.

simulate the interactive behavior of input devices early inthe design process. The central idea is to support interactive
sketching so that next stage of constructing a more refined
prototype, one that relies on computational materials, is better
informed.
4.1.1. Sketching with interactive behaviors
Thinking in terms of computational materials instead of
hardware prototyping is an important step in supporting
sketching in an organic design. We position computational
materials as having a similar role to that of the automotive
industry’s use of clay to create a replica-size model of a car;in this context, it is a material used to create a high production
value prototype. Although computational materials are criticalto construct organic designs, their use requires some technicalexpertise and physical instrumentation of prototypes, somethingthat could be inhibiting when ideating in the very early stagesof a design.
When sketching an organic design, and by that we meana rough and early-stage exploration of a potential design,
not unlike an architect’s sketch of a building, we argue thatinteraction should be decoupled from its material embodiment.
Computational materials should be mimicked as interactive
behaviors, ones effortlessly imbued in a physical prototype. This
carefully removes the material challenges of designing OUIs
and makes it easier to sketch them.

Figure 7. A designer can work with a range of materials using
SketchSpace. This includes everyday objects (cup, paper, box),
deformable surfaces (CD cover, tape) and malleable material(modeling clay).

To imbue interactive behaviors in a passive physical prototype
this way, SketchSpace imbues passive prototypes with input
sensing using Microsoft’s Kinect camera. Mathematically analyzing the depth sensing using techniques from computationaland differential geometry allows SketchSpace to infer a broad
set of inputs, ranging from touch events on the object’s surface and surrounding workspace, orientation, position, motion,
proximity and, among others, shape deformations (see Figures 6
and 7). A designer uses these inputs to prototype as if it had functionality, including mapping sensor values to interactions using
embodied gestures and projecting dynamic interactive content

Interacting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

Figure 6. SketchSpace tracks and projects a texture on a roughly
sculpted mouse made of modeling clay. The icons on the top, left andbottom correspond to virtual sensors. The icons on the right are fordisplaying images and adding or removing virtual buttons. In counterclockwise from the ‘is below icon’ (down arrow): is above, is right of,is left of, is rotated right, is rotated left, is tilted up, is tilted down, istilted left, is tilted right, is near, is higher than, is deformed (measured
as the distance changed in curvature features), is grasped (measured inamount of hand pixels obscuring the object) and button tapped.

140

David Holman et al.

5. TOWARD HYPER-CONTEXTUALIZED DESIGN
If interaction is truly everywhere and tools such as SketchSpace,
Tactiletape, OmniTouch and TDR help designers rapidly
explore or prototype OUIs using computational materials, what
design objectives should guide them? Each form, from Gummi’s
mobile concept device to Benko et al.’s sphere, has its own

nuances. Across these forms, however, one question is relevant:should the interface be multi-purposed (like a Swiss army knife)or specialized (like a classic Wustof cooking knife)? In short, it
depends.
One thing is clear: there is a new opportunity to build highly
specialized devices, ones that can use organic principles toseamlessly contextualize interaction in form. This distinction
moves away from the approach of the computer as a generic
tool (Buxton, 2007; Vertegaal, 2011) and introduces cautious
specializations in the interface. By doing so, the interactive
behavior of some types of organic design will only express a
few essential actions, ones that are subject to their form factor.
What exactly is this category of specialized organic devices?
There is a distinction to be made for multi-purposed form
factors. Certainly, there will always be a class of devices
with both highly purposed functionality and organic principles,
Nokia’s Kinetic being one example (Kildal et al., 2012). The
brick shape of modern smart phones, even when deformable, hasabsorbed a camera’s function and a GPS’s mapping capability.
Even Snaplet (Tarun et al.. 2011) suggests an OUI that canbe repurposed depending on its context, transforming its shape
to flat, to be drawn on; when cylindrical, used as an armband
music player. A similar example is found in Gummi’s flexible
interaction of credit card-sized device.
However, as organic design embarks on a multitude of new
interactive shape and form, there will be a greater opportunity
for form factors that are highly contextualized in their shape. For
example, imagine a toothbrush covered in a thin-film interactive
skin. In an effort to promote better oral hygiene, it might useful
to indicate on the brush’s handle that it is being held properly
and the correct pressure is being applied. In any case, the shapeof the brush commands a unique way of interacting with it, onethat is not so easily transformed and absorbed by another form
factor.
To accept computational things is to assume a scenario where
many passive everyday objects could be imbued with interactive
potential. We argue that, for this new set of form, factors should
be designed by hypercontextualizing their interface. A bank cardwith a thin film display would show your balance and recent
activity, but it would not be used to write an email or browse
the web. It might have an interactive map on the back, but its
interactive behavior would be limited to helping you perform
mobile financial transactions or finding a nearby branch. In asimilar example, a reusable water bottle might subtly glow when
it is near a water fountain, only if it is almost empty. A high-end
kitchen knife might indicate when the incorrect pressure andangle is applied during a cut.
This stringent minimalism has clear benefits: it limits therange of possibilities a user encounters using highly purposed
types of organic interfaces. It also leverages in situ interface
design, in that physical manipulations of a computational thingcan be more precisely mapped, even made more usable, whenfunctionality is limited and excels at doing only one or two
things. Keeping the back of the bank card as just a map, as

Interacting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

on it, all without finding, attaching and working with a collection of physical sensors. The designer works hands-on by tapping mapping icons that are projected on their workspace table
and, using physical manipulations of an object in situ, to specify when and how these mappings trigger interactive behavior.
Using a tool like this frees the designer from hardware constraints and helps them ideate and quickly explore interactive
sketching.
However, exposing these virtual sensor streams to a
designer poses the challenge of mapping the lower-level input
patterns to meaningful forms of interaction in a way that is
intuitive and non-technical. SketchSpace extends a designer’snatural manipulation and defines input mappings through the
combination of tabletop sensor icons, embodied gesturing andsimple speech commands. First, there are 15 sensor icons that
represent mapping rules available to the designer. Each of theseconvey a useful property of the prototype’s input behavior thatcan be toggled on or off by tapping its icon. In Figure 6,
the down arrow indicates an ‘is below’ condition indicatingwhen the prototype is below the threshold value, the mapping
rule should trigger. The value of a virtual sensor is saved asa mapping rules, one set by physical performing an object’s
behavior and issuing a speech command, and is used to trigger
a position change in a preset set of interface mockup images
the designer provides beforehand. In general, this approach
frees the designer’s hands for bimanual asymmetric interaction
and quickly generates interaction rules for each of the virtual
sensors, ones that are mapped to rough, sketched content.
Although early user experiences with SketchSpace’s efficacy
as a sketching tool have pointed us toward key design decisions,such as projecting icons on a designer’s tabletop and supporting
embodied mapping gestures, it has been challenging to deploy
SketchSpace in an industrial design studio. At present, wehave deployed SketchSpace in a semi-guided scenario, during auniversity-level design studio focused on designing interactive
systems, ones that recreated the objects from DisplayObjects(Akaoka and Vertegaal, 2010) to form a baseline comparison
and inform our active development for broader deployment.
Students were able to rapidly sketch the dynacan and paper
computer, but the interactive sphere interface was limited as
projector could not fully render on its surface until it was placed
flat on the desk, making it impossible to rotate in place. This,of course, could be addressed through the use of one or more
projectors, or even a wearable device like OmniTouch (Harrison
et al., 2011) that can leverage its ability to imbue surfaces withinteraction for the sketching of an organic design.

Design of OUIs: Shape, Sketching and Hypercontext

6.

CONCLUSIONSThis paper argues that the best thing that could happen to a userinterface design is for computers to stop being technologicaldevices and start being more like real everyday things. Not
only does this effect the OUIs we create, but the tools andmethodologies we use to design them too. In future, their
hypercontextualized interfaces will feature the same type ofskins we find on products today, but extended with a minimalistic and carefully selected interactive behavior. Designing
this category of OUIs to excel at perhaps one or two functions
at a time radically simplifies the design of the user interface
and ensures that the ‘computer’ dissolves from the forefront.
Interactive hardware will be a mere commodity to the industrial
designer, to point that it looks and feels like any other designmaterial. This, naturally, accelerates the need for new toolsthat allow designers to sketch using computational materials.
This is a turning point for a user interface design, one thatchallenges us to understand how to best design in this newworld of everyday computational things.

REFERENCES
Abbot, E.A. (1884) Flatland: A Romance of Many Dimensions. Seely& Co., United Kingston.
Akaoka, E. and Vertegaal, R. (2010) DisplayObject: Prototyping
Functional Physical Interfaces on 3D Styrofoam, Paper, or
Cardboard Models. In Proc. ACM TEI’10, pp. 49–56. ACM Press,
Cambridge, MA, USA.
Ballagas, R., Memon, F., Reiners, R. and Borchers, J. (2007) iStuff
Mobile: Rapidly Prototyping New Mobile Phone Interfaces for
Ubiquitous Computing. In Proc. ACMSIGCHI’07, pp. 1107–1116.
ACM Press, Cambridge, MA, USA.
Benko, H., Wilson, A.D. and Balakrishnan, R. (2008) Sphere: Multitouch Interactions on a Spherical Display. In Proc. UIST’08, pp. 77–
86. ACM Press, Cambridge, MA, USA.
Buxton, B. (2007) Sketching User Experiences: Getting the Design
Right and the Right Design. Morgan Kaufmann, San Francisco, CA.
Cassinelli, A. and Ishikawa, M. (2005) Khronos Projector. In Proc.
SIGGRAPH’05, Article 10. ACM Press, Cambridge, MA, USA.
Dijkstra, R., Perez C. and Vertegaal, R. (2011) Evaluating Effectsof Structural Holds on Pointing and Dragging Performance with
Flexible Displays. In Proc. CHI’11, pp. 1293–1302. ACM Press,
Cambridge, MA, USA.
Girouard, A., Tarun, A. and Vertegaal, R. (2012) DisplayStacks:
Interaction Techniques for Stacks of Flexible Thin-film Displays.
In Proc. ACM CHI 2012, pp. 2431–2440. ACM Press, Cambridge,
MA, USA.
Han, J. (2005) Low-cost Multi-touch Sensing Through Frustrated Total
Internal Reflection. In Proc. UIST’05, pp. 115–118. ACM Press,
Cambridge, MA, USA.
Harrison, C., Benko, B. and Wilson, A. (2011) OmniTouch: Wearable
Multitouch Interaction Everywhere. In Proc. 24th Annual ACM
Symp. User Interface Software and Technology (UIST ’11),
pp. 441–450. ACM Press, New York, NY, USA.
Hartmann, B., Klemmer, S.R., Bernstein, M., Abdulla, L., Burr,
B., Robinson-Mosher, A. and Gee, J. (2006) Reflective PhysicalInteracting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

opposed to including a web browser or any other ‘useful’ applications, means that the user simply flips their credit card to see
the nearest branch. This minimizes application switching, displaces a cumbersome smartphone and embeds this information
in a predictable locale, one that is tied to physical starting pointof their mental model and banking experience.
It is worth pointing out that hypercontextualization alreadyexists in some consumer devices. Amazon’s Kindle, forexample, excels at being just one thing: a digital representationof a book. Once this boundary is exceeded, by trying to surf
the web on its E Ink display, its hypercontext dissolves andthe simplicity of its interaction metaphor breaks down. In thecontext of organic design, there is the potential to create many
new types of hypercontextualized devices. Although someinterfaces will be highly purposed, there will be a new class oforganic interfaces that, via computational materials, leverages
hypercontext to more tightly couple interaction, form andfeedback, like the examples of the organic toothbrush or waterbottle, in interface design.
Thinking about hypercontextuality applied to an interface
design is only a first step. How designers should go about
representing it in three-dimensional interface design requires
elaboration. Unlike the user-centered design process, settlingon the functions that are most critical to embody in an OUI
is not necessarily linear or even formulaic; shape is transformed in early design iterations and since the interface is so
closely bound to it, it requires constant reevaluation and designreflection. This is not to say that the traditional methods ofuser-centered design cannot still inform the design. However,the context in which an interface exists and the shape it occupies have a much deeper effect than it did with a traditional
design. Thinking back to our reusable water bottle, a designermust deeply consider its size, material, relationship to ownerand, among other aspects, the environment in which its identity
plays out. Traditionally, this is the domain of industrial design.
However, designing a water bottle made from a computational
material must also account for the ways in which the bottle
can be augmented with an interactive form. A starting pointis a simple question grounded in McLuhan’s law of extension(McLuhan, 1964): of the functions (or experiences) this waterbottle exhibits (or should exhibit), which of them would beenhanced, intensified or made possible by imbuing them withinteractive behavior? Perhaps calmly revealing the temperatureof the water after it is filled up would impact a person’s choiceof water fountain. Or maybe tapping the bottle at a certain
spot would reveal how much water the owner has consumed.
For even simple things, the possibilities quickly exhaust themselves. It is at the discretion of the designer to decide how theinterface will be hypercontextualized or if it should be at all.

141

142

David Holman et al.

Prototyping Through Integrated Design, Test, and Analysis. In
Proc. ACMUIST ’06, pp. 299–308. ACM Press, Cambridge,
MA, USA.

Rekimoto, J. (2002) SmartSkin: an Infrastructure for Freehand Manipulation on Interactive Surfaces. In Proc. CH’02,
pp. 113–120. ACM Press, Cambridge, MA, USA.

Holman, D. and Benko, H. (2011) SketchSpace: Designing Interactive
Behaviors with Passive Materials. In Extended Abstracts of CHI
2011, Vancouver, BC, Canada. ACM Press, Cambridge, MA, USA.

Rosenberg, I. and Perlin, K. (2009) The UnMousePad: an Interpolating
Multi-touch Force-sensing Input Pad. In Proc. SIGGRAPH’09,
pp. 1–9. ACM Press, Cambridge, MA, USA.

Holman, D. and Vertegaal, R. (2008) Organic user interfaces: designingcomputers in any way, shape or form. Commun. ACM, 51,
48–55.

Roudaut, A., Pohl, H. and Baudisch, P. (2011) Touch Input on Curved
Surfaces. In Proc. CHI’11, pp. 1011–1020. ACM Press, Cambridge,
MA, USA.

Holman, D. and Vertegaal, R. (2011a) Designing Everyday
Computational Things: Why Industrial Design Will Be the New
Interaction Design. In Industrial Design Society of America
Education Symposium, New Orleans, LA, USA. IDSA Press.

Sato, T., Mamiya, H., Koike, H. and Fukuchi, K. (2009)
PhotoelasticTouch: Transparent Rubbery Tangible Interface Using
an LCD and Photoelasticity. In Proc. UIST’09, pp. 43–50.

Holman, D., Vertegaal, R., Altosaar, M., Troje, N. and Johns, D. (2005)
PaperWindows: Interaction Techniques for Digital Paper. In Proc.
ACM CHI 2005, Portland, OR, pp. 591–599.
Hudson, S. and Mankoff, J. (2006) Rapid Construction of Functioning
Physical Interfaces from Cardboard, Thumbtacks, Tin Foil and
Masking Tape. In Proc. ACM UIST’06, pp. 289–298. ACM Press,
Cambridge, MA, USA.
Ishii, H. and Ullmer, B. (1997) Tangible Bits: Towards Seamless
Interfaces between People, Bits and Atoms. In Proc. CHI 1997,
pp. 234–241. ACM Press, Atlanta.
Ishii, H., Lakatos, D., Bonanni, L. and Labrune, J. (2012) Radical
Atoms: Beyond Tangible Bits, Toward Transformable Materials. In
ACM Interactions 19, 1 January and February 2012, pp. 38–51.
ACM Press, Cambridge, MA, USA.
Kildal, J., Paasovaara, S. and Aaltonen, V. (2012) Kinetic Device:
Designing Interactions with a Deformable Mobile Interface. In
CHI’EA 12, pp. 1871–1876. ACM Press, Cambridge, MA, USA.
Knörig, A., Wettach, R. and Cohen, J. (2009) Fritzing—a Tool for
Advancing Electronic Prototyping for Designers. In Proc. ACM
TEI’09, pp. 351–358. ACM Press, Cambridge, MA, USA.
Lahey, B., Girouard, A., Burleson, W. and R. Vertegaal. (2011)
PaperPhone: Understanding the Use of Bend Gestures in Mobile
Devices with Flexible Electronic Paper Displays. In Proc. CHI
2011. ACM Press.
Lee, J.C., Avrahami1, D., Hudson, S.E., Forlizzi, J., Dietz, P.H.
and Leigh, D. (2004) The Calder Toolkit: Wired and WirelessComponents for Rapidly Prototyping Interactive Devices. In
Proc. ACM DIS’04, pp. 167–175. ACM Press, Cambridge,
MA, USA.

Schwesig, C., Poupyrev, I. and Mori, E. (2004) Gummi: A Bendable
Computer. In Proc. CHI’04, pp. 263– 270. ACM Press, Cambridge,
MA, USA.
Song, H., Benko, H., Guimbretière, Izadi, S., Cao, Xiang, Hinckley, K.
(2011) Grips and Gestures on a Multi-touch Pen. In Proc CHI’11,
pp 1323–1331.
Stevenson, A., Perez, C. and Vertegaal, R. (2011) An Inflatable
Hemispherical Multi-touch Display. In Proc. TEI’11, pp. 285–288.
ACM Press, Cambridge, MA, USA.
Tarun, A. Lahey, B., Girouard, A., Burleson, W. and Vertegaal,
R. (2011) Snaplet: Using Body Shape to Inform Function in
Mobile Flexible Display Devices. In Extended Abstracts CHI’11,
pp. 329–334.
Vertegaal, R. (2011) The (re)usability of everyday computational
things: why industrial design will be the new interaction design.
ACM Interact. 18, 38–41.
Villar, N. and Gellersen, H. (2007) A Malleable Control Structure forSoftwired User Interfaces. In Extended Abstracts of ACM TEI’07,
pp. 49–56. ACM Press, Cambridge, MA, USA.
Villar, N. et al. (2009) Mouse 2.0: Multi-touch Meets the Mouse. In
Proc. UIST ’09, pp. 33–42. ACM Press, Cambridge, MA, USA.
Vogt, F., Chen, T., Hoskinson, R. and Fels, S. (2006) A Malleable
Surface Touch Interface. Technical Sketch. In Sketches andApplications at SIGGRAPH’04, p. 36.
Weiser, M. (1991) The computer for the 21st century. Scientific Am.,
265, 94–104.
Weiss, M. Voelker, S., Sutter, C. and Borchers, J. (2010) BendDesk:
Dragging across the Curve. In Proc. ITS’10, pp. 1–10. ACM Press,
Cambridge, MA, USA.
Wigdor, D. and Wixon, D. (2011) Brave NUI World: Designing Natural
User Interfaces for Touch and Gesture. Morgan Kaufmann, San
Francisco, CA.

McLuhan, M. (1964) Understanding Media: The Extensions of Man.
McGraw Hill, New York.

Wilson, A. (2010) Using a Depth Sensor as a TouchSensor. In Proc.
ITS’10, pp. 69–72. ACM Press, Cambridge, MA, USA.

Piper, B., Ratti, C. and Ishii, H. (2002) Illuminating Clay: a 3D Tangible Interface for Landscape Analysis. In Proc. CHI’02,
pp. 355–362.

Wimmer, R. and Baudisch, P. (2011) Modular and Deformable TouchSensitive Surfaces Based on Time Domain Reflectometry. In Proc.
UIST 2011, pp. 517–526.

Interacting with Computers, Vol. 25 No. 2, 2013

Downloaded from http://iwc.oxfordjournals.org/ at Queen's University on May 1, 2013

Holman, D. and Vertegaal, R. (2011b) TouchTape: Low-Cost Touch
Sensing on Curved Surfaces. In Extended Abstracts of UIST 2011,
October 2011. ACM Press.