© ACM, 2010. This is the author's version of the work. It is posted here by permission of ACM for your personal use.
Not for redistribution. The definitive version was published in the Proceedings of CSCW'10.

Pictionaire: Supporting Collaborative Design Workby Integrating Physical and Digital Artifacts
Björn Hartmann1, Meredith Ringel Morris2, Hrvoje Benko2, Andrew D. Wilson2
1 – University of California, Berkeley, EECS Dept.
2 – Microsoft Research, Redmond, WA
bjoern@eecs.berkeley.edu{merrie, benko, awilson}@microsoft.com
ABSTRACTThis paper introduces an interactive tabletop system thatenhances creative collaboration across physical and digital
artifacts. Pictionaire offers capture, retrieval, annotation,and collection of visual material. It enables multiple
designers to fluidly move imagery from the physical to thedigital realm; work with found, drawn and captured
imagery; organize items into functional collections; andrecord meeting histories. These benefits are made possible
by a large interactive table augmented with high-resolution
overhead image capture. Summative evaluations with 16
professionals and four student pairs validated
discoverability and utility of interactions, uncovered
emergent functionality, and suggested opportunities fortransitioning content to and from the table.
Author Keywords

Surface computing, design tools, tangible interaction.
ACM Classification Keywords
H5.3. Information interfaces and presentation (e.g., HCI): Group andOrganization Interfaces – Computer-supported cooperative work.

General Terms

Design, Human Factors
INTRODUCTION

Discussion in design is almost always tied to concrete
artifacts: designers use sketches, photographs, and physicalprototypes as examples to structure conversation andcollaboration [5]. Creative work moves freely acrossphysical and digital boundaries, but today’s computing
infrastructures are often oblivious to this co-existence [7].
This note hypothesizes that interactive tables are
particularly suited to support and enhance group design
work when they enable co-habitation of physical and digital
artifacts on the table. We contribute Pictionaire, a tabletop
system for design teams that introduces interactions for
capturing physical artifacts and organizing their digitalcopies. Pictionaire augments multi-touch input with highresolution digital still image capture above the table,
enabling teams to work with digital images and text, paperPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee.
CSCW 2010, February 6–10, 2010, Savannah, Georgia, USA.
Copyright 2010 ACM 978-1-60558-795-0/10/02...$10.00.

Figure 1. Pictionaire supports design collaboration acrossphysical and digital artifacts.

based sketches, and physical artifacts (Figure 1). We also
contribute an evaluation that highlights tradeoffs between
physical and digital interactions for creative group work.
Pictionaire extends related work in computing
environments for co-located design teams. The multi-user
Shared Design Space [3] focuses on high-resolution digitaldrawing but does not offer support for capturing images ofphysical artifacts. PostBrainstorm [2] is an interactive muralthat supports image capture on a neighboring table.
Pictionaire extends the PostBrainstorm research by merging
capture and display space, thereby allowing newinteractions. The Designer’s Outpost [8] allows coexistence of paper Post-it notes and digital ink on a wall
display, but does not offer support for multiple
simultaneous users or other materials.
Interactive tabletops have also introduced interactions
across bits and atoms outside the design domain. Wellner’s
Digital Desk [11] targeted the co-existence of paper anddigital information for office work. Augmented Surfaces
integrated personal computing devices and projectedtabletops [9]. Data Tiles [10] introduced referenced
projection on top of physical objects, treating these objects
as containers for digital content. Pictionaire builds upon thiscontainer metaphor and focuses specifically on supporting
design team meetings.
In the following scenario enabled by Pictionaire, interaction
designers Max and Mia create a UI walkthrough of a touch
screen camera during a meeting. Max first searches forimages of digital cameras online by typing a query on awireless keyboard (Figure 2A). He selects a suitable
photograph from the search results and passes the image to© ACM, 2010. This is the author's version of the work. It is posted here by permission of ACM for your personal use.
Not for redistribution. The definitive version was published in the Proceedings of CSCW'10.

Figure 2. Using Pictionaire to draw a UI walkthrough. A: Search for images. B: Drag result to table. C: Snap image to physicalwhiteboard. D: Capture copy of annotated image. E: Organize images as a collection.

Mia on the table (2B). To sketch on top of this photo, Mia
drags the image onto a letter-sized whiteboard tablet (2C).
The image reorients and resizes to fill the tablet. Mia pulls
the tablet toward her and, with a dry-erase marker, sketchesa touch screen interface. To capture this first frame of her
walkthrough, she makes a digital copy of her drawing witha drag-off gesture (2D). As she sketches additional UI
screens, Max organizes her captured images by droppingthem into a timeline container, which orders the images
sequentially (2E). Max plays the walkthrough as an
animation, and he and Mia reflect on the design.
We next discuss how Pictionaire enables this scenario.
FORMATIVE DATA AND DESIGN GUIDELINES

Based on interviews with interaction designers and two
observations of a product design team, we formed four
guidelines that influenced Pictionaire’s design: n Offer alarge, standing height form factor to approximate designers’
workbenches and meeting tables. o Support work with aheterogeneous set of digital and physical artifacts such as
web images, sketch books, and form prototypes. p Focus
on discussion, ideation, and sketching; detailed design
mostly takes place after meetings. q Support input inmanners media designers are accustomed to (pens fordrawing; standard input devices for application control).
TABLE INFRASTRUCTURE

Pictionaire runs on a custom-built interactive table thatmeasures 120cm × 180cm × 91cm (Figure 3). Graphics are
top-projected with two tiled XGA projectors (1024×1536
pixels; 21.3 dpi). Touch input and object location aresensed through diffuse IR illumination with two tiled VGA
cameras (640×960 pixels; 13.3 dpi). A high-resolution

Figure 3. The Pictionaire table offers touch and device input,
top projection, and high-resolution image capture.

digital still camera centered above the table captures photosof the entire table on demand (4272×2848 pixels; 59.3 dpi).
A Windows Vista workstation runs vision processing andapplication logic based on a custom version of theMicrosoft Surface SDK. To enable efficient text input,
precise pointing, and remote interaction, Pictionaire
incorporates interaction techniques for mice and keyboards
on interactive surfaces [4].
INTERACTION TECHNIQUES

Pictionaire supports group design sessions thoughinteraction techniques for searching & tagging, physical-todigital transitions, digital-to-physical transitions, remotehighlighting, and image organization.
Searching & Tagging

Designers often seek inspiration by searching for example
imagery on the web [5]. With Pictionaire, wirelesskeyboards can be used to initiate such queries, and to entertext annotations. To switch between these functions, soft
buttons are projected next to the keyboard when it is placedon the surface. When a user begins to type, a text input box
that displays the entered text appears on the table justbeyond the top edge of their keyboard (Figure 2A). When asearch is executed, the text entry box is replaced with acontainer showing a grid of images returned by the search(2B). Users can transfer images from this container onto thetable through direct touch or mouse input. Typedannotations can be overlaid on any digital item on the table.
Physical to Digital: Drag-Off

Paper-based sketches were frequently used alongside digital
images in our formative studies. To capture such sketches,users can place sheets or notebooks anywhere on the table.
Objects are distinguished from finger touches by their sizeand shape. When an object is recognized, digital handles
are projected around the corners of the object (Figure 4,left). Users can then drag-off from the projected handle
onto the table area to create a digital copy: the item isphotographed by the overhead camera, and a digital imageat 1:1 scale is created. Copies can be manipulated in usual
multi-touch fashion (via multi-point translate, rotate, andscale). When the user executes a drag-off gesture, thecamera takes an image of the entire table and crops thatimage using a rotated bounding box calculated from theellipse model of the object’s contact area supplied by our
input SDK (Figure 4, right).

© ACM, 2010. This is the author's version of the work. It is posted here by permission of ACM for your personal use.
Not for redistribution. The definitive version was published in the Proceedings of CSCW'10.

physical realm. Users place an object, e.g., a form
prototype, on the table and capture a digital copy. After
activating highlighting mode, pointing to features on thedigital copy creates a projected highlight on thecorresponding location on the physical artifact (Figure 6).
This technique enables designers to direct attention without
obscuring the view for others.
Figure 4. Recognizing a paper drawing. Left: Virtual handles
projected around a sheet of paper. Right: Ellipse (blue) andbounding box (red) of paper as calculated by our vision system.
Digital to Physical: Copy-and-Annotate

Sketches are not only brought to meetings, they are also
produced during meetings to illustrate ideas on the fly, andare then taken from meetings for later review as graphical“to do” lists. To support composing digital and physicalimagery, digital images can be snapped to drawing surfaces
on Pictionaire. To preserve the analog feel and resolution ofpens, Pictionaire uses letter-sized dry-erase whiteboards forthis function. Designers drag a digital image onto one of thevirtual handles of such a tablet (Figure 2C). The projected
image is resized and reoriented to fill the tablet (preserving
aspect ratio). After sketching on the image, designers thendrag out a new digital copy which contains both the originalgraphic and the added annotation. To capture thecomposition, Pictionaire currently takes a single photo. To
increase fidelity, one could photograph only the marks on
the tablet and digitally compose this image with thesketched-upon image, or use digital pen technology [3].
Digital pens provide high-resolution input but constrain the
choice of drawing instrument and surface. Using paper
notebooks instead of whiteboards enables use of Pictionaire
as a light table to create hand-traced copies of digital
images (Figure 5).
Remote Highlighting

During discussion it is beneficial to direct group attention todetails of an artifact. The Spotlight technique [6] achieves
this function on digital images by projecting a circular
highlight on demand. Pictionaire extends Spotlight into the

Figure 5. Using Pictionaire to trace images in sketchbooks.

Organizing Digital Material

To organize visual material into meaningful aggregates,
Pictionaire offers collection containers that hold multiple
images. These containers enable grouping, sorting, andanimation of image sets through three views: n a freeformlayout, where individual images can be freely moved,
scaled, and rotated; o a grid view where images are tiled tomaximize table real estate; p a flipbook view that offers atimeline and playback of images as an animation (Figure2E). The flipbook view supports the practice of presenting
interface walkthroughs of UI prototypes and enables stopmotion animation.
Capture and later replay of table sessions can help attendees
recall meetings and report them to others. Pictionaire offersa capture function that photographs the entire table.
Histories can be assembled by manually triggering capture
or by setting an automatic interval. A special flipbook
container is dedicated to browsing these session histories.
USER EXPERIENCES

How well does Pictionaire support the needs of real design
teams? 16 design professionals evaluated our table in openended sessions. Five tested individually; others in groups oftwo to four. The professionals tried out Pictionaire’s
features and were asked to relate how it could be used intheir work. In addition, eight graduate students completed a
30-minute brainstorming task in pairs. The student pairs
were asked to brainstorm ways to re-design a handheld
game controller. All participants were invited to complete
an online survey — 13 responded.
Patterns of Use

Brainstorm pairs alternated between individual work
(generating ideas independently; one person sketches while
the other conducts a web search) and shared work
(discussing an issue together). By providing multiple
independent sets of input devices and tablets, Pictionaire
supports switching between these two work styles well.
One designer commented that placing tablets and keyboards
onto a shared display prevents individuals from “hiding”
behind opaque laptop screens, a common meeting problem.
In the brainstorming task, the most frequently performed
operations were capture of whiteboard sketches (22occurrences in 4 sessions) and web search for images (20occurrences), corroborating our motivation that work acrossboth digital and physical artifacts should be supported. An
example of a team photographing a physical controller, thenannotating the resulting photo is shown in Figure 7.

Figure 6. Projecting light onto a physical object by pointing
on its digital copy. Red arrow indicates highlight.

Our professional participants imagined a range of different
uses for Pictionaire that leveraged image capture,© ACM, 2010. This is the author's version of the work. It is posted here by permission of ACM for your personal use.
Not for redistribution. The definitive version was published in the Proceedings of CSCW'10.

ojected image,, as someerase or lighten paarts of a proj
particippants attempted.

Figure 7. Annootations on a
photo of a gamme controller.

Figure 8. Haand included innphoto pointss out a feature.

All groups in the brainstormb
taskk faced overlyy clutteredocan bee shuffled,
tables at some poinnt. Physical objects
stackedd, and pushed out of the wayy. When physiical objects
touch digital objectss during clean up on Pictionnaire, thosemove as well. But digital objects cannot nott influencephysical objects, nor can they move each other. It
I thus took
particippants longer too clean up digiital messes thaan physicalmessess. This issue couldcbe addressed by mim
micking thereal worldw
through simulated coollisions or thhrough an
automaatic cleanup fuunction. To avoid interferencce between
digital and physical objects, technniques for obsttacle-aware
projecttion [1] could beb applied to Pictionaire.
P
FUTUR
RE WORK: WO
ORK PROCESSS INTEGRATIION

Figure 9. Im
mpromptu imag
ge capture of a laptoplscreen.

ssometimes in unforeseenuway
ys. Three partiicipants included
thheir hands in the images tak
ken — to poinnt out importaantpparts
of a draw
wing, or to dem
monstrate a grrip on an objeect(Figure 8). Onee participant photographedph laptop screen
his
u
using
Pictionairre as a fast meethod of sharinng an image wiitha teammate (Figure 9). Havinng sharing technology availabble
innstantly was more importtant to him thantquality ofrreproduction.
TTwoparticipan
nts in separate sessions createdsstop-motion
annimations by firrst drawing onn pieces of papeer,thhen rearranginng these pieces between image captures.
The post-use suurvey elicited advantages off Pictionaire ovverTccurrent
analogg and digitaal practices. Most frequeentrresponses
werre that Pictionaire promotees collaborativvewwork(7x); it offfers a form faactor large enouugh for a varieetyo work mateerials (6x); itof
i enables raapid capture &aannotation(5x)); and allows projecting lighht onto physiccaloobjects(3x). Participants providedpfreee-form answerrs,w
which
we cateegorized. Lower counts mayy not mean thhatpparticipants
fouund fault with an aspect, onlyy that they choose
too focus their coomments elsew
where.
N
Navigatingthee Boundary Be
etween Bits an
nd Atoms

While the geneeral flow of skeetching with phhysical tools anndWthhen capturingg digital copiees was well understood, thheddivision
betweeen bits and atom
ms was not alw
ways clear to ouropparticipants.
Ouur participants’ mental modeel appeared to beb““projected
lighht can be man
nipulated by toouch.” Howeveer,
P
Pictionairecann only sense touch on the tabble surface itseelf,n on objects. Thus, drag-offf gesture can only be initiated
not
f
from
digital handleshon coorners of whitteboards. In thhesstudy,
some participantspattempteda(unssuccessfully) toinnitiate drag-off
ff gestures on th
he whiteboard tablet.
Another limittation of capturingAcdraawings througghpphotography
iss that annotatiing a whiteboard with a daarkmmarkeror draw
wing on white paperpcan onlyy make the inkedppartsof the capptured image daarker. Thus it is
i not possible toPictionnaire introducces a set of techniques thhat, taken
togetheer, enable and encourage co--habitation of digital andphysical artifacts to facilitate collaborative creaative work.
mprovement
The moost frequently mentioned sugggestion for im
was too integrate Pictionaire into larger work processes.
Particippants appreciaated that the taable specificallly supportsco-locaated meetings.. This focus alsoaimplies thhe need forindividdual preparatioon and follow--up away fromm the table.
Efficieent import and export of conteent is thereforee required.
REFER
RENCES
1. Cottting, D. and Gross,
G
M. Inteeractive environnment-aware
dispplay bubbles. Proceedings
Pof UIST 2006, AC
CM (2006),
2455-254.
2. Guiimbretiere, F., Stone, M., anda
Winograd,, T. Fluid
inteeraction
withhigh-resolutiionwall-size displays.
Prooceedings of UISST 2001, ACM (22001), 21-30.
3. Halller, M., Brandl, P., Leithinger, D., Leitner, J., Seifried, T.,
andd Billinghurst, M.
M Shared Design Space: Skettching Ideas
Usinng Digital Pens and a Large Augmented Tabletoop Setup. In
Advvances in Artificiial Reality and Tele-Existence.
T
2
2006.
4. Harrtmann, B., Morris, M.R., Bennko, H., and Wilson,
W
A.D.
Auggmenting Interaactive Tables withw
Mice & Keyboards.
Prooceedings of UISST 2009, ACM (22009), 149-152.
5. Herrring, S.R., Chang, C., Kranttzler, J., and Bailey,
B
B.P.
Gettting inspired!: understandinguhhow
and why exxamples areusedd in creative deesign practice. ProceedingsP
off CHI 2009,
ACM
M (2009), 87-966.
6. Khaan, A., Matejkaa, J., Fitzmauricce, G., and Kurrtenbach, G.
Spootlight: directinng users' attenntion on largee displays.
Prooceedings of CHII 2005, ACM (2005), 791-798.
7. Klemmer, S.R. Inttegrating physiccal and digital interactions.
i
EE Computer 38,, 2005, 111-113..
IEE
8. Klemmer, S.R., Neewman, M.W., Farrell,
F
R., Bileezikjian, M.,
andd Landay, J.A. The
T designers' outpost:oa tangibble interface
for collaborative web
w site design. Proceedings
Pof UIST 2001,
ACM
M (2001), 1-10.
9. Rekkimoto, J. and Saitoh,S
M. Augm
mented surfaces: a spatiallyconntinuous work space for hybridd computing ennvironments.
Prooceedings of CHII 1999, ACM (1999), 378-385.
10. Rekkimoto, J., Ullmmer, B., and Obaa, H. DataTiles: a modular
plattform for mixed physical annd graphical interactions.
i
Prooceedings of CHII 2001, ACM (2001), 269-276.
11. Welllner, P. Interaacting with paaper on the DigitalDesk.
D
Com
mmun. ACM 36, 7 (1993), 87-966.