Pen + Touch = New Tools
Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington,
Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton
Microsoft Research, One Microsoft Way, Redmond, WA 98052{kenh, mpahud, nicolec, jrodenh, awilson, benko, bibuxton}@microsoft.com; yatani@gmail.com
ABSTRACTWe describe techniques for direct pen+touch input. We
observe people’s manual behaviors with physical paper andnotebooks. These serve as the foundation for a prototype
Microsoft Surface application, centered on note-taking andscrapbooking of materials. Based on our explorations weadvocate a division of labor between pen and touch: the pen
writes, touch manipulates, and the combination of pen +touch yields new tools. This articulates how our systeminterprets unimodal pen, unimodal touch, and multimodal
pen+touch inputs, respectively. For example, the user canhold a photo and drag off with the pen to create and place a
copy; hold a photo and cross it in a freeform path with the
pen to slice it in two; or hold selected photos and tap one
with the pen to staple them all together. Touch thus unifies
object selection with mode switching of the pen, while themuscular tension of holding touch serves as the ―glue‖ thatphrases together all the inputs into a unitary multimodal
gesture. This helps the UI designer to avoid encumbrancessuch as physical buttons, persistent modes, or widgets thatdetract from the user’s focus on the workspace.
Author Keywords

Pen, touch, gestures, tabletop, tablets, bimanual, systems
ACM Classification Keywords

H.5.2 Information Interfaces and Presentation: Input
General Terms

Human Factors
INTRODUCTION

Direct interaction with displays is rapidly becoming one ofthe primary means by which people experience computing.
This has led to renewed interest in devices with multi-touch
input, as well as pen input, manifest in form factors ranging
from hand-helds, slates, booklets, desktops, table-tops, andwall displays. The iPhone, iPad, Project Courier [29] (seealso [11,20]), Wacom Cintiq, Microsoft Surface, andSmartboard are, respectively, examples of each. But only
rarely do systems support both modalities, much less the
combination of the two for simultaneous pen + touch
interaction [6,38,39]. Meanwhile many efforts continue toconsider either touch input in isolation, or pen input in
isolation. Yet if dual-mode digitizers with both pen andtouch represent the future of displays [15,21], the presence
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee.
UIST’10, October 3–6, 2010, New York, New York, USA.
Copyright 2010 ACM 978-1-4503-0271-5/10/10....$10.00.

of both modalities should significantly alter our perspective
on how to most effectively design for multi-touch, as wellas the pen. Hence simultaneous pen+touch is a nascent
topic that demands further study and exploration.

Fig. 1.

Pen writes, touch manipulates, pen + touch = new tools.

This argues for a holistic systems-oriented approach, rather
than focused evaluation of one-off techniques [17]. Our
experience is that trying many ideas [8]— some good, someperhaps bad in retrospect, and some intentionally chosen tohighlight conflicting conventions or thorny design decisions
rather than hide them— is an excellent way to draw out
nuances and gain insights into novel input modalities (e.g.
[22,25,37]). Much of the value of pen+touch interaction
seems to accrue from the workflow— that is, the flow ofinteraction across a series of interrelated subtasks— rather
than from individual techniques considered in isolation.
Exploring a rich application context offers a realistic
perspective of how pen+touch influences UI design issues
and trade-offs. We inform our design with an observational
study of 8 people, and discuss feedback from 11 test users,
but do not present a detailed evaluation in this manuscript.
We discuss Manual Deskterity, a ―scrapbook‖ application[20,23,28] for the Microsoft Surface which serves as a
virtual designer’s studio: that is, part digital desk and part
digital moleskin notebook [8,19]. While a few other efforts
do consider pen and touch (e.g. [6,38,39]), we contribute ananalysis that significantly advances the topic by identifying
nine key design considerations of pen+touch. We also
contribute observations of people’s use of physical paperand notebooks: there is a rich set of deeply rooted behaviors
that people exhibit when working with pen, paper,clippings, pages, and books, with both stylus and barehanded manipulation, that differentiate between the roles of
pen and touch. We leverage this to define the primitives of
our interaction language, and to design pen+touch gesturesthat go well beyond pens and paper to support novel tools.
From an input-theoretic standpoint a key insight of thiswork is that for multimodal pen+touch interactions, a single
direct-touch action (holding an object) integrates threesteps: (1) object selection; (2) mode switching [27,31] of
the pen from ink strokes to contextual commands that act inreference to the selected object; and (3) the phrasing
together of multiple pen and touch inputs into a unitarycommand [9]. Furthermore the muscular tension of holding
an object serves as salient kinesthetic feedback [34] thatunambiguously ties all of these things together. This isachieved without recourse to physical buttons, persistentmodes, or permanently visible widgets that compete withthe workspace for screen real estate and the user’s attention.
RELATED WORK

Systems with direct input often employ only one of pen(e.g. [2,25,33]) or touch (e.g. [14,36,37]). Yet an earlier
generation of devices, such as the Palm Pilot (1996),
supported use of either pen or touch. However, thetechnology could not differentiate pen vs. finger contact.
But a new generation of digitizers is emerging that cansense multi-touch while distinguishing pen from touch, thus
enabling pen AND touch rather than pen OR touch [15].
Several research efforts explore the combination of pen andtouch. Yee [39] uses single-touch + pen input to support
panning a canvas and drawing. Wu [38] describes two
examples of pen and touch gestures. Brandl [6] explores
bimanual pen + multi-touch techniques that assign the pen
to the preferred hand and touch to the nonpreferred hand.
We instead tease apart pen vs. touch, preferred vs.
nonpreferred hand assignment, and unimanual vs.
bimanual interaction, among other factors. For example,
we consider unimanual usage contexts where the userinterleaves pen and touch interactions with the preferredhand, and we explore a wider vocabulary of novel pen +touch gestures that afford compound transactions.
Cohen discusses the complementary role of pen gesturesand natural language [12]; he treats multimodal input with aprobabilistic approach [13]. We instead treat pen+touch in amanner that affords deterministic state-machine-driven
GUI’s. Also, because pen and touch are both manual input
modalities, the nuances of how the two complement one
another are more subtle, and we must overcome a legacy ofdesigns that have treated pen or touch interchangeably.
Guiard observes that the hands cooperate [18], so thequestion is not Which hand is better? but rather ―What is
the logic of the division of labor between the hands?‖
Likewise, in our research we ask: What is the logic of thedivision of labor between pen and touch in UI design? Ourgestures also build on Guiard’s observation that thenonpreferred hand frames the action of the preferred hand.
T3 explores indirect bimanual input with a multi-button
puck and stylus input [26], including airbrushing along a
french curve held in the nonpreferred hand. We explore
analogous pen + touch techniques to draw along a
straightedge, but our techniques employ entirely button-freedirect input. Studies of two-handed input have emphasized
how bimanual interactions afford compound tasks, such as
navigation-selection tasks [10,26]. The design approach of

chunking and phrasing [9] proposes aggregation of atomictasks into higher level ―cognitive chunks.‖ We show how
pen + touch offers a new arsenal of techniques to support
such compound tasks, without recourse to buttons [27].
Frisch describes a set of user-elicited gestures where users
often treat pen and touch interchangeably [16]. As a result,the gestures contain many ambiguities. Which pen, touch,or pen+touch gestures should a system support, and why?
User-defined gestures yield insights but cannot be taken too
literally: people have difficulty envisioning how to use newmodalities of which they have little or no experience [17].
Indeed, our exploration of Manual Deskterity convinces usthat if each input modality offers complete coverage of allinteractions, it quickly robs the combination of pen andtouch of much of its vigor. Differentiating between pen andtouch, rather than treating them interchangeably, offers aconsistent and rich designed input vocabulary. Nonetheless
we build our gestures and techniques on a vocabulary ofnaturally occurring manual behaviors, as shown below.
DESIGN STUDY USING A PAPER NOTEBOOK

We conducted a design study with a paper notebook to gain
insight as to how people work with physical tools andpieces of paper. We asked each participant to illustrate theirideas for a hypothetical short film by pasting andannotating clippings in a paper notebook. To simulate aslate computer where the user could move between pages,
we provided a paper notebook (7‖ x 10‖ when opened) as
the authoring space. Users sat at a table stocked with tools
such as pens, glue, tape, and scissors, as well as print-outsof inspirational materials (20 visually salient web pages).
Eight people (6 male, 2 female, age 9–53 years, all righthanded) participated in the study. We taped the sessions
and looked for patterns in how users worked, gestured, held
objects, or structured their working space. From thisexercise we observed behaviors (B0-B9) that informed
specific gestures as well as design attributes of our system.
B0. Specific Roles. Participants wrote with the pen, andarranged clippings with their fingers, often using both
hands (Fig. 2a, 2b). Users did not move objects withthe pen. While seemingly obvious, we call thesebehaviors out here because they conflict with the onehanded, unimodal methods of interaction that typicallyencumber users on devices such as Tablet PC’s.
B1. Tuck the Pen. People tucked the pen between theirfingers to interleave writing and moving clippings
around with their preferred hand (Fig. 2a), or flipping
pages (Fig. 2f). Participants were remarkably adept at
rapidly interleaving the pen and fingers in this manner.
B2. Hold Clippings. People temporarily held clippings inplace with a finger of their nonpreferred hand (Fig. 2a).
B3. Hold while Writing. Participants tended to hold aclipping with their nonpreferred hand while writing
about it with the pen. This was often the case even if
the clipping was glued into place, or if the annotations
were to the side of the clipping (Fig. 2b).

B4. Framing. A common hand posture was to hold thumband index finger to ―frame‖ clippings (Fig. 2b). Thisappeared to be a behavior to focus attention on asource object and reference annotations to it, rather
than one strictly necessary to hold it in place.
B5. Scraps. Participants cut clippings from the inspirational
materials while holding a sheet in their nonpreferredhand, above the notebook. Users at times adopted theresulting scraps when they fell onto the page (Fig. 2c).
B6. Extended Workspace. Participants arranged theworkspace with the notebook proximal to their body,while reaching above it to access tools and materials. A
surround of clippings, unused pages, and tools formed
an extended workspace beyond the notebook. (Fig. 2d).
B7. Piling. All users formed piles of ―interesting‖ clippings
while holding other items in the nonpreferred hand.
B8. Drawing along Edges. While not a common behavior,a couple of people did employ clippings as a constraint
for the pen, to draw a border around an item (Fig. 2e).
B9. Hold Page while Flipping. Participants kept their placein the notebook by holding a thumb or fingers of thenonpreferred hand on a page while flipping to distal
pages with the preferred hand (Fig. 2f).

DESIGN CONSIDERATIONS FOR PEN AND TOUCH

Informed by the above observations, as well as current
research and industry trends in direct input devices, we can
now show how the design space of pen + touch interaction
raises many interrelated factors and considerations. The
most obvious, of course, is the factor of pen vs. touch input
itself. Without feigning completeness, the following tableausummarizes the interaction properties shared by pen andtouch of particular relevance to our discussion here. We do
not characterize such properties as ―pros‖ and ―cons,‖ ashas been attempted elsewhere [6], to accentuate our belief
that almost any property of a device can be advantageous ininteraction design. For example, accidental palm contact
can seen as a problem, or it can viewed as an opportunity tocorrectly orient menus for each of multiple table users [7].
PROPERTY
PEN
TOUCH
Contacts

1 point
A single well-defined point.

1-10+ contact regionswith shape information.

Occlusion

Small (pen tip)
But hand still occludes screen.

Moderate (“fat finger”) to
Large (pinch, palm, whole hand)

Precision

High - Tripod grip & lever arm Moderate
for precision, writing, sketching.

Hand

Preferred hand

Either hand / Both hands

Elementary Tap, Drag, Draw Path,
Inputs
Crossing

Tap, Hold, Drag, Pinch, 2-finger
Hold (thumb + index, Fig. 2b)

Intermediary

None: Bare-Handed Input
Nothing to unsheathe or lose.
No lever arm. No buttons.

Mechanical Intermediary
Takes time to unsheathe thepen. Pen can be forgotten.

Acquisition High (first use: unsheathe pen) Low
Time
Moderate on subsequent uses: No mechanical intermediary totuck pen between fingers (B1). acquire.

Activation
Force

Non-ZeroZero. Contact area (a proxy for
Tip switch/ minimum pressure. pressure) often can be sensed.

False
InputsPalm Rejection (while writing) Midas Touch Problem
Palm triggers accidental inputs, Fingers brush screen, finger
fingers drag on screen, etc.
rests on screen while holding it.

Fig. 3.

Fig. 2.

Behaviors observed during design study. See text.

The observations above contribute naturally occurring
behaviors, with both pen and bare-hand manipulation, thatexhibit a clear differentiation between the roles of pen andtouch. Terrenghi also observes a rich set of bimanual
behaviors resulting from physical manipulation of clippings[35]. These stand in contrast to the results of user-elicited
gestures for pen and touch [16], which might otherwise lead
us to treat many pen and touch inputs interchangeably. Our
observations suggest it would be mistaken to do so. By thesame token, our goal is not to mimic paper. Pen+touchshould go beyond paper, but the best foundation for
pen+touch gestures likely lies in behaviors that people
already exhibit when working with pen, paper, and books.

Summary of design properties for pen vs. touch.

We can now briefly map out the main design considerations
that we believe the design space of pen+touch raises:
1. Pen vs. Touch: The type of input, as summarized above.
2. Differentiated vs. Interchangeable task assignment: Fora given input task, does the interface differentiate
between pen and touch, or are they interchangeable?
3. Nonpreferred vs. Preferred hand assignment: Whichdevice is assigned to which hand, and under whatcircumstances?
4. Unimodal vs. Multimodal input: Inputs can be articulatedusing either unimodal pen, unimodal touch, or
multimodal (simultaneous) pen+touch. How are each ofthese used to best advantage in an interface design?
5. Mobile vs. stationary use: Desktop, table, and wall direct
input devices are necessarily stationary, but form-factors
such as slates and booklets [11,20] transition between
mobile and stationary use (the user can pick them up, or
put them on a desk). How can pen+touch support acommon conceptual model to enable gracefuldegradation between stationary and mobile usage?
6. Unimanual vs. bimanual interaction. For what tasks andusage scenarios does the user interact using one hand,versus both hands? Can the user complete core tasks
one-handed? Does the interface leverage the fullpotential of human hands, as well as simultaneous pen +touch input, when working with both hands?
7. Interleaved vs. Simultaneous inputs: Does an interaction
require simultaneous pen+touch, or can users perform it
by interleaving a series of touch-pen-touch interactions,with the user tucking the pen as needed?
8. Ink vs. Command mode: Are movements of the device
left as an ink stroke (or finger smudge)? Or are they
instead interpreted by the system as a gesture (or direct
manipulation) that acts on the underlying object?
9. Elementary inputs vs. Phrases: Does the design consider
just the compatibility of the input device to individualinteractions, or does it support a workflow acrossmultiple tasks, or subtasks, that therefore afford phrasing
together of actions into compound tasks?
We do not have answers to all of these questions. But ourenumeration of these considerations builds on thecontributions of Brandl [6] to further map out the design
space of pen and touch. Furthermore we can now articulate
how our approach falls into place along these dimensions: Unimodal pen vs. touch have differentiated effects in
the interface (per observation B0). We assign ink modeto the pen, while multi-touch gestures articulatecommands: the pen writes, and touch manipulates.
 By tucking the pen (per B1), the user can efficiently
interleave pen and touch inputs with the preferred handfor mobile, unimanual usage scenarios; Designing core tasks for unimanual touch serves
mobility while also enabling stationary bimanualinteraction that instead assigns these tasks to thenonpreferred hand, thus supporting more efficientinteraction as well as advanced gestures; These benefits are derived while retaining thepossibility of bimanual interactions with multimodal
pen and touch: pen + touch = new tools.
 For simultaneous pen+touch, holding a finger on an
object (B2) integrates object selection, transition to
gesture mode, and the phrasing together of multiple penor touch inputs into a compound gesture [9,34]. That is,pen strokes in reference to an object while the userholds it are recognized as gestural commands, rather
than ink strokes that would otherwise mark the object.
With the foundation for our approach now set, we turn ourattention to our prototype application and the novel
interaction techniques that it supports.
APPLICATION SCOPE AND MOTIVATION

Manual Deskterity is intended primarily as a researchvehicle to explore pen + touch, which we believe has manypotential applications. Nonetheless we emphasize practical
tools for note-taking and mark-up noted by previous work
(e.g. [20,23,28]). For example, several papers emphasize
reading [1,24] in conjunction with writing, annotation,
selecting, copying, arranging, and aggregating objects, bothin digital [2,38] and paper-based knowledge work

[4,24,35]. In this context reading as a pure ―consumption‖
activity is a myth; it is inseparable from the annotation andcreation of new work artifacts.
Furthermore, jotting notes, or annotating a book, take place
in a surround of tasks and activities that embody the user’s
higher level workflow [1,24]. What other information isread and cross-referenced with the sheet of paper the user iscurrently writing on? How does the user structure andaccess this workspace? Our observations of user’s extended
workspaces (B6), the frequency of reaching into the space
above the notebook to retrieve objects (also B6), and thefacility with which users flip back and forth between distal
pages of a notebook (B9) serve as specific examples. Since
hand and stylus are closely tied to how people work withbooks and documents, to realize pen and touch techniquesalong these themes Manual Deskterity supports both wholescreen and dual-screen [11,20] views. Pen+touchtechniques may afford digital workspaces with a feel closer
to the informal way that people actually work with sets ofactive documents [1,24] and scraps of information [28].
Our design study suggests a few additional behaviors ofinterest in the context of simultaneous pen + touch, such as
holding items while acting on them with the pen (B2, B3,
B4), cutting operations (B5), and employing clippings as aconstraint for the pen (B8). These and the considerations
discussed above motivate the features that we elected to
explore using pen, touch, and pen+touch interactions.
Implementation

We use a Microsoft Surface with a custom infrared LED
pen, activated during contact via a tip switch. Our pen ismuch brighter than hand/finger contacts, so it is trivial toidentify as the brightest spot in the image. The software is
written in C# with WPF and the Microsoft Surface SDK.
A potential limitation of pen+touch is the ―palm rejection‖
problem: the user may rest his hand on the screen whilewriting, potentially leading to unintended operations. We
treat touches with a large contact area as incidental. This issufficient for prototyping pen + touch techniques, butrobust handling of incidental contacts remains an important
problem for future work. Our approach does not solve it.
CORE TASKS: PEN WRITES, TOUCH MANIPULATESWe drive our core interactions by touch interactions
including zooming, flipping pages, moving and selecting
objects, and creating new objects (e.g. digital post-it notes).
For writing, only the pen produces ink strokes (although insome contexts the finger ―smears‖ colors, as discussed
later). Notwithstanding the exception of finger-painting, forthese core tasks the pen writes, and touch manipulates,
period. This makes the entire canvas available for
immediate annotation with the pen, while pan/zoom, pagenavigation, and object manipulation are also immediately
available via touch, without any explicit mode switches. Of
course, we say the pen writes as shorthand for any
graphological act that leaves a mark, whether it comprises
sketching, annotating, cross-hatching, or handwriting.

Manipulating, Zooming, and Selecting Objects

The user can arrange objects such as photos and post-it
notes on the canvas via direct manipulation (single touchdrags objects, while pinching objects rotates and scalesthem). Two-finger pinch on the canvas zooms the entire
page, allowing detailed edits. Since these core tasks are
driven by unimanual touch, users can perform these actions
with the preferred hand (e.g. for mobile usage scenarios), orwith the nonpreferred (while the preferred hand remains
ready to go with the pen) for bimanual usage scenarios.
The user can select multiple objects via finger-taps that
incrementally add individual objects to the current
selection. Dragging an item that is already selected drags it
and all other selected items while maintaining their relative
spatial relationship. Dragging an item that is not alreadyselected drags only that item; this enables multi-touchdragging of items to multiple different locations.
Pen always writes vs. pen drags selected items: In oursystem, the pen (when used as a unimodal input) always
writes on items, even if they are selected. Beyond recourse
to our observation of specific roles (B0), why is this so? We
experimented with moving selected objects with the pen to
support a precise dragging mode (as proposed by Brandl[6]). However, we found that users would frequently intend
to write on a selected item with the pen, but drag it bymistake. The feedback for selection is a prominent pinkhighlight that surrounds the object. But this feedback isdelivered through the visual channel, so it is not particularly
salient to the user [34]. How might this be improved? Wealso tried dragging the item only when the user holds it withthe finger. The muscular tension of holding a finger on theobject provides more salient feedback, and as a resultdragging with the pen feels more natural to users. However,this is now a multimodal pen+touch gesture, which has
many more possibilities; we will return to these shortly.
Interchangeable inputs in common controls: On selection, aradial menu appears at the upper right corner of objects,
with options such as copy, delete, clear annotations, etc.
Initially, our radial menus required use of the pen, but wefound that all test users expected this menu to be operable
via touch as well. Although we have so far advocateddifferentiation between the roles of pen and touch, this is agood example of a context where users clearly expect pen
or touch inputs to have the same effect in the interface. We
still advocate the principle that the pen writes and touch
manipulates, but the lesson here is that an exception must
be made for common controls such as menus, close boxes,
sliders, and buttons that represent limited spatiallymultiplexed contexts where pen and single-touch inputs
should indeed be interchangeable.
PEN + TOUCH YIELDS NEW TOOLS

Next, we explored how the expanded input vocabulary
afforded by the combination of pen and touch can support
new tools. Our design uses a few primitives for all pen +touch operations. The richness of the gestures arises from
how the primitives are combined. For the pen the primitives

we use are tap, drag-off, crossing, or drawing a stroke. For
touch, we employ single-finger tap, single-finger hold (asseen in our design study, behavior B2), holding with thumband forefinger (B4), and crossing. We do not implement all
combinations, but rather support a sufficiently rich set ofoperations, with semantics that map well to our application
domain, to illustrate the expressiveness of our approach.
The pen+touch techniques described below all use fingersof the nonpreferred hand hold an item, while the pen acts inreference to the item. This builds on the tendency that we
observed for users to hold clippings with the nonpreferredhand while making pen markings in reference to them (B3),and it also corresponds to Guiard’s principle that thenonpreferred hand sets the frame of reference for the
preferred hand [18]. Thus, we construct gestures that allow
for non-physical digital effects, yet remain grounded inpeople’s naturally occurring behaviors with physical paper.
Stapler: Grouping Items into a Stack

To support piling (observation B7), we implemented
stapling items into a stack. The user can finger-tap-selectmultiple items, and then staple them together by holding an
item and tapping it with the pen. The item that the user
performs the ―staple‖ (hold-and-tap) gesture on moves tothe top of the stack, so that it represents the entire pile.
Tap-selecting a series of items and stapling them together
enables quick tidying of a messy work surface into a few
piles. This transaction separates the identification of theitems to stack from the decision of which item should
become the representative item on the top of the resulting
stack. Performing the pen+touch gesture on the
representative item keeps the user’s attention focused on it,
at exactly the moment the user makes this decision, which(based on our informal user studies) corresponds well withusers’ mental model of the task. While piling has been
explored by other systems (e.g. [2]), what is unique here is
the facility of creating stacks, with a representative image
on top, via a dedicated pen+touch gesture. Test users rated
this as one of their favorite tools, and particularly liked theability to easily specify which object is on top of the stack.
We could have used tapping with a second finger forstapling. Why use the pen instead of touch? Although our
test users did not ask for this, the question once again
highlights the tradeoff of differentiated vs. interchangeable
pen and touch inputs. Because touch requires zero contact
force, we have observed that accidental taps are common.
That is, holding items and tapping with a finger is prone toaccidental activation in a way that holding plus tappingwith the pen is not. Using pen and touch for this gesture
therefore feels more distinct to the user, while also more
effectively leveraging the strengths of each input modality.
X-acto Knife: Cutting Items (and Tearing Items)

The user can turn the pen into an X-acto knife by holding
an object and fully crossing it with the pen. That is, the pen
stroke starts outside the object, crosses through the interiorof the object, and finishes on the exterior of the object. Thepen stroke within the item can follow any path, allowing
intricate cuts if desired (Fig. 4). When the pen exits theitem, both the cut and the scrap piece appear on the page,
following the real-world action where we observed scraps
falling onto the work surface (design study B5). Users
found the technique natural and compelling, even though
not all saw a need for it in their daily work. Most users
liked the scrap cuts, and expected both pieces to remain: as
one user asked, ―How would it know which half I want?‖

Fig. 4.

Example X-acto cuts created by one test user.

To probe the semantics of pen vs. touch in analogous
gestures, we also implemented tearing items by holding an
object with a single finger, and then crossing the item withanother finger. This tears the item along the line connecting
the entry and exit points of the finger. This is similar to
cutting, but produces a different visual affordance. This
technique demonstrates how touch can sometimes be used
to sneak a nuance of expression into a transaction, byapplying a different look or different default commandparameters. On the other hand, in our system this precludes
using the touch gesture for a different command, such as
layering [14]. Should the semantics of analogous pen+touch
vs. touch+touch gestures be similar or contrasting?
In this regard, our test users did not perceive tearing photos
as a particularly useful operation; for example some users
suggested that tearing could support different semantics,
such as deleting the object. But they also did not suggestthat both versions of the gesture should provide the same
X-acto cut behavior. The lesson is that users perceive
different semantics; most people do not expect analogous
pen+touch and touch+touch gestures to do the same thing.
Carbon Copy: Drag-Off with the Pen

The user copies an object by holding it with a finger (B2)
and then ―peeling off‖ a copy with the pen (Fig. 7). This
gesture is similar to Frisch’s copy gesture [16], but here we
identify the interaction pragmatics as well as why thisgesture differs from a Copy command on a context menu,
for example. Here we seen an example of how the userelicited gestures methodology can yield fertile ground forsuggesting plausible gestures (so long as we keep in mind
that users are not designers [17]).
Once the pen drags away by a minimum distance, a semitransparent copy of the object appears attached to the pen.

As the pen continues to drag, the object becomes opaque.
The user continues dragging the object to ―paste‖ it at thedesired location. Annotations on the object are also copied.
Our system also supports Copy via the object’s context
menu. Drag-off with the pen phrases together the entire
transaction (select, copy, and drag to final position) into asingle cognitive chunk [9] via the muscular tension of thenonpreferred hand holding down the original item. We
found that this corresponds well to users’ mental model of
duplicating items– they not only want to copy the item, butalso place the duplicate at a particular location. By contrast,
Copy from the context menu divides this select-copyposition transaction into multiple steps, enabling onehanded copying at the cost of syntactical complexity [9].
Holding an item and peeling off a copy with the pen is agood example of a pen+touch technique that ostensibly
violates the principle that the pen writes and touch
manipulates, because here the pen drags the copy.
However, those principles apply to pen or touch as
unimodal inputs. The transaction is consistent with theprinciple that guides our multimodal gestures: pen + touchyields new tools. Our test users found the gesture naturaland effective. Furthermore it is grounded in people’snaturally occurring behaviors with physical paper, such as
holding an item and making pen strokes in reference to it(Guiard [18], and our design study observations B2, B3).
Ruler: Using an Object as a Straightedge

The user can employ an object as a straightedge by holding
down the object with the thumb and index finger, like theframing gesture observed in our design study (B4, Fig. 2b).
The user can then stroke along the object with the pen
constrained to its border, as inspired by observation B8.
When two fingers come down on an object, an animation
starts that increases the transparency of the item and adds a
dotted line around its border. Informal test users who tried
early versions of the system suggested that items should
become mostly-transparent in this manner. This enables
users to see the relationship of the straightedge to theunderlying surround of other objects and strokes.
The Ruler uses content as its own tool, introducing a subtle
duality between content and tool into our system. We could
add a dedicated ruler, but this would necessitate acquiring
the ruler before drawing a straightedge. Using an object as
its own straightedge also supports sketching techniquessuch as adding a drop shadow or outer glow to an item. To
be clear, however, we are not arguing that the systemshould not include a ruler tool; rather, we are arguing thatusing an object as its own straightedge has interaction
design properties that a dedicated ruler tool does not.
Composition of the Straightedge with Cutting

To illustrate how our interaction design allows multiple
interactions to be composed together into phrases, we
implemented X-acto cutting along straightedges (Fig. 5).
The user finger-tap-selects the photo to cut, and then uses
the two-finger thumb and index finger grasp on anoverlapping item to establish a straightedge. The user thenstrokes the pen across the selected photo along thestraightedge to cut it. The user can even cut around thecorner of the overlapping item. Composition of techniques
is possible because tapping provides a way to define acollective scope [25] of more than one object, and because
holding phrases together multiple pen and touch inputs intoa single chunk that the system interprets as a command [9].

system, by default the pen draws freeform ink strokes.
While the user is drawing a pen stroke, he just holds afinger on it to turn the remaining portion of the stroke into atape curve. Lifting the finger from the stroke immediately
reverts to the default freehand ink state. While thetechnique seems intriguing, we have not yet user-tested it.

Fig. 7.

Tape: Users can fluidly mix ink, straight lines, & curves.

Color Palette: the Finger Shadow

Fig. 5.

Cutting along an edge (a); the scrap falls on the page (b).

Brushing: Using a Photo as Its Own Tool

Our system uses pen+touch to enable the user to employ
content on the page as a brushing tool. If the user holds aphoto and strokes the pen on the canvas, the pen stroke uses
the photo as a brush effect (Fig. 6). While the user
continues to hold, each subsequent pen stroke is mapped to
a common viewport within the photo so that the user may―rub out‖ different portions of the photograph (see video).
This technique is similar to the Art History Tool in Adobe
Photoshop, but our technique can produce some creative
effects without the complexity of modes, parameters, andlayers required to use the Photoshop Art History Tool.

Fig. 6.

We have also experimented with the Finger Shadow (Fig.
8), which uses a finger-tap on the canvas to bring up inplace commands. This accesses additional commands,
including color pots (Fig. 8). In early demonstrations of oursystem we observed that most people expect touching thecolor pots to enable smearing of colors with the finger. To
probe this issue further, in this context we intentionally
break our design rule for unimodal inputs that the pen
writes and touch manipulates. To some degree, this echoes
our earlier observation that common controls should allow
use of either pen or touch. Yet here, pen and touch are notcompletely interchangeable either: the pen produces a crisp
ink stroke, while the finger smears colors onto the page(Fig. 8, right). To prevent finger painting from becoming aheavyweight mode, each successive finger paint stroke
appears fainter until ―all the ink has been rubbed off‖ theuser’s finger (Fig. 8, right). This echoes how artists workwith physical media such as charcoal sketching.

A brush effect can be applied by holding a photo.

This technique provides another example of the facility
with which the user can hold an item to produce a modethat is specific to that object, or that uses the object as aparameter. It also illustrates how the application of thatmode can include one or more strokes (applications) of thetool while the nonpreferred hand holds the state.
Tape Curve: Holding a Pen Stroke as a Drawing Tool

Tape drawing is a familiar technique for illustrators: any
craft store carries ―artist tape for curves.‖ To produce astraight line the artist pins the end of the tape with the left
hand, unrolls a length of unfastened tape with the righthand, and then slides the left hand along the tape to adhere
it. Alternatively, to draw curves, the artist pivots theunfastened length of tape in the right hand while the left
hand follows the tape. Balakrishnan [3] observes that
corresponding digital tape exhibits an intriguing interaction
property: it supports both straight lines and variable radius
curves within the same tool, without switching modes.
Balakrishnan’s implementation is in ―tape drawing mode‖by default, and requires physical buttons [3]. But in our

Fig. 8.
Left: Dragging from the Finger Shadow (here with thepen) opens the menu. Right: Finger painting using the color pots.

With finger painting, we have to face a genuine design
dilemma: when ink remains on one’s finger, should it bepossible to finger paint on top of objects? In our system, theanswer is no. Touching an object to select or move it (ortouching other controls, such as flipping pages) always
takes precedence over finger painting. Otherwise, a more
explicit means to ―stop‖ finger painting and return to thedefault behavior that touch manipulates would have to be
introduced, and we did not wish to do so at this point.
NEW “TOUCHES” ENHANCE PEN+TOUCH WORKFLOW

Finally, we explore some techniques that extend theinteraction states that can be supported with multi-touch
interactions. This enhances the workflow of interleaved
touch-pen-touch interactions, whether articulated with both
hands, or with one hand by alternating pen and touch.
Creating New Objects via the Bezel Menu

Earlier we observed that users bring in new materials fromabove (B6). Creating a new object implies making

something out of nothing: how does one ―directly
manipulate‖ objects that are not on the screen?
Furthermore, most of the time one wants to create and place
an object [25]. Standard answers to these questions
consume screen real estate (e.g. tool palettes), or input
device state transitions (e.g. buttons, tap-and-hold) thateither aren’t available for direct input, or could be betterused for other purposes. To address this challenge wedevised a novel finger-activated bezel menu that extends theconcept of bezel-crossing gestures [30], while adaptingfrom marking menus the concept a graceful novice-toexpert transition with self-revealing gestures [25].

attention– much as is the case for real document navigation.
All the user has to do is hold the current page with thenonpreferred hand. As long as the user holds it, he cannavigate to other pages (Fig. 10b) and later flip back bysliding the nonpreferred hand towards the spine of the book
(Fig. 10c, d). Users liked flipping back and forth,particularly for comparing pages, but sometimes lifted by
accident (thus losing the ―held‖ page). This interaction was
only implemented for the dual page view because themetaphor seemed to make the most sense there.

Fig. 9.

Fig. 10. Holding a page. (a) Hold with left hand; (b) Flip throughpages with the right hand.; (c) Sweep inwards with the left hand to(d) flip back to the original “held” page.

Creating new objects via the bezel menu. See text.

The bezel menu is visible only as a semi-transparent strip
along the top edge of the screen (Fig. 9a). To access it, theuser slides a finger across the bezel, which starts to pull out
the bezel menu (Fig. 9b). With the bezel menu open, theuser may then tap on an icon to create a new object such as
a post-it note. This is the self-revealing, ―novice‖ mode of
making a selection. The bezel menu then hides itself again.
In the expert gestural mode, once a user is familiar with thelocation of commonly used items in the bezel menu, he canperform a continuous finger-drag that crosses through theitem and onto the canvas (Fig. 9c) to create and place an
object (such as a digital post-it) in a single transaction. This
aspect of the technique particularly resonated with test
users. The user can then immediately annotate the post-it,
thus supporting a tight create-place-annotate workflow.
Navigation to Pages, Books, & the Extended Workspace

To flip the page, we again employ a bezel gesture: the user
swipes across the top right corner to go to the next page, oracross the top left corner to go to the previous page. The
user can flip through multiple pages by starting at thecorners, and then dragging downward to scrub through thenotebook. A page curl in these corners provides a subtle
visual affordance for page flipping. Page flipping responds
only to touch; the pen can still write in this area.

Fig. 11. Pinching across the pages in 2-up view supports
semantic zooming to the “desk view” extended workspace.

Flipping between Distal Locations

Cross-Screen Pinch: Zoom to Extended Workspace

To support observation B9, Hold Page while Flipping (Fig.
2f), we implemented a technique whereby the user can holdtheir place in the dual-screen view of the digital notebook
by resting a finger of the nonpreferred hand near the marginof the page (Fig. 10a). This enables the user to keep theirplace in a document with very little effort or conscious

What about navigation to the ―extended workspace‖surrounding the notebook (B6)? In dual-screen view, ourapplication supports cross-screen pinch to semanticallyzoom to a ―desk view‖ that serves this role (Fig. 11). Here,the user can arrange books, documents, and photos. The
user can leave post-it notes or other visuo-spatial cues ascontextual reminders that implicitly indicate what to donext when resuming a task [4,24]. This aspect of theextended workspace particularly resonated with users.
Note that a pinch gesture that spans the two pages offers agesture that is distinct from both zooming within a page
(two-finger pinch on a page) as well as scaling of individual
objects on the page (via two-finger pinch of the object).
Thus, like bezel gestures, cross-screen gestures offer
another ―escape hatch‖ in the interaction design languagethat enables simple and unambiguous expression of
additional interactions with gestures that correspond well tothe semantics of the desired operation– here, semantic
zooming to an extended workspace.
Summary Discussion of “New Touches”

One could argue that these are just touch techniques thathave little to do with pen and touch input. Yet whilewriting, the secondary tasks of creating new objects,
flipping back and forth between pages, and navigating to
the extended workspace can all help retrieve, compare, andcreate task-relevant information. Because these fluidly
support the primary task of writing and annotating, thesetouch techniques afford tight interleaving of secondarytasks (via the nonpreferred hand) with the primary task
(writing and annotation with the pen). This helps to ―keep
the user in the flow‖ of concentrated work. It is also tellingthat our desire to support this workflow helped to spark theidea for techniques that extend direct touch interfaces,
which comes full circle to our belief that the presence ofboth pen and touch modalities can alter our perspective on
how to most effectively design for touch itself.
DISCUSSION & HIGH-LEVEL REACTIONS FROM USERS

Our multimodal pen+touch gestures use touch to holditems, in combination with one or more pen strokes that act
in reference to an item that the user is holding. Thisapproach enforces a strong notion of phrasing [9] in the
resulting interface design: there are no persistent modes,
but rather tool use is always tied to holding an object with
the nonpreferred hand. There is no possibility of getting
stuck in a tool mode with the pen, nor is there ever aquestion of how to return to the default action of drawing
ink strokes on the page. Once the user releases objects,
drawing on the page with the pen always leaves ink strokes.
This approach is akin to nonpreferred-hand mode switching[27], but requires no physical button and thus readily scales
to a plurality of modes [32]. It also offers an additional
advantage: object selection is integrated with the modeswitch itself when the user’s hand touches down on an
object on the display. Hence a unique design property of
pen+touch is the facility with which it can support modes
and tools specific to particular objects on the screen. Thatis, we contribute the insight that object selection, modeswitching, and phrasing together multiple inputs are all
seamlessly integrated in this class of technique.
While our emphasis in this paper has not been on
evaluation, we founded our approach on a design study ofpeople’s naturally occurring behaviors, and we have tested
the techniques with 11 test users. Users may not have needfor some specific tools, and although our current prototype
includes all of these techniques, it is not our expectationthat future systems would need to include all of thesetechniques, as some have been added for pedagogicalreasons so as to draw out the nuances of how pen and touchcomplement one another.
Thus, our current gesture set is more complex than a ―real‖
production system would support. The combination of pen+ touch does not automatically extract us from the classicaltrap with gestural interfaces: each additional gesture that
one adds to a system proportionally reduces the value of all
the other gestures. However, the distinctness of using pen
versus touch, coupled with the highly contextual nature ofhuman memory, does seem to modestly increase thenumber of gestures that a system can include, versus a
system based solely on touch, for example. Nonetheless thisbelief remains to be tested by formal experiments.
Users found our approach to combined pen and touch
interaction appealing. They quickly formed habits aroundthe general pattern (hold touch + gesture with pen) common
to most of our gestures. As one user commented, ―I didn’t
think about switching between pen and touch at all… it justkind of happened…‖ Another commented that ―the way itworks is just like the way I already work in my notebook.‖
But clearly, we have a designed gesture set that we would
not expect users to ―guess‖ without guidance. As one usercommented, ―I wouldn’t have guessed the gestures workthat way, but once I tried it, it felt pretty natural.‖ This
does bring us to one general weakness of the prototype:currently the multimodal pen+touch gestures are not self
revealing, nor in most cases is there sufficient feedback of
the current action of the tools until after the user finishes
articulating a gesture. In some cases it would be difficult to
add feedback at the initiation of a gesture because theresulting effect (e.g. stapling vs. drag-off-copy) cannot beknown until the pen is lifted or starts to move. This issue isexacerbated in our prototype because our pen has no hoverstate, which precludes the possibility of showing the current
function before the pen touches down. Nonetheless webelieve existing self-revelation techniques, such as
GestureBar [5] or Highlighter Hints [23], would be suitablefor integration with our prototype.
As noted by a reviewer, the concept of an object underlies
many of our gestures, but also suggests a more nuanced
hierarchy of contextual actions on object types. The canvas
is a manipulable element (it zooms and pans) as well as acontainer for pieces of documents (e.g. post-its, photos).
These pieces serve as the background for smaller entities,
such as notations upon a post-it. Indeed, users expect themarks upon such pieces to move with the piece; ourimplementation even goes so far as to split ink strokes atthe boundaries of the pieces to maintain this metaphor.
Furthermore, ―system‖ objects such as menus and tools

play by different rules that the pieces. Even this category isblurred (as it is in real life), for example by our use ofphotos as straightedges. This hints that perhaps a deeper
theory or guiding principles for contextual actions, which
act upon on a hierarchy of object types, may exist.

8.

CONCLUSION AND FUTURE WORK

12.

Despite rapture with the iPhone (and now iPad), multitouch is not the whole story. Every modality, including
touch, is best for something and worst for something else.
The tasks demanded of knowledge workers are rich andhighly varied [1,4,24], and as such one device cannot suit
all tasks equally well. One’s finger is no more suited forsigning a contract, or drawing a sketch on a napkin, than isa pen for turning the page of a book, or holding your place
in a manuscript. With the addition of the pen, userinterfaces afford creation of new ideas, rather thanunbounded consumption of content produced by others.
We have advocated an approach where the pen writes,
touch manipulates, and the combination of pen+touchyields new tools. This articulates how our system interprets
unimodal pen, unimodal touch, and multimodal pen + touch
inputs, respectively. The space of pen and touch, while
potentially complex, is also sufficiently rich that it offers
the opportunity to craft new user experiences that are
uniquely well suited to how people naturally work with pen
and paper—without being beholden to physical mimicry of
paper at the same time. It remains to prove that ourapproach scales from a demo to a full-blown application,
nor have we yet demonstrated how these rich techniques
enhance the effectiveness and user experience of lessglamorous applications, such as working with a spreadsheet. We believe that they can and will, but that is a longshot from actually doing so. As well, despite our hopes andprojections as to how well these techniques will work onother form factors, the fact remains that we have not yet
done those tests. There is still work to do.
ACKNOWLEDGEMENTS

Special thanks to Georg Petschnigg, Andrew Allen, Jon
Harris, Scott Erickson, Sabrina Boler, Tito Pagan, Joel
Jacobs, and Yasser Asmi for ideas, insights, & discussions.
REFERENCES
1.
2.
3.
4.
5.
6.
7.

Adler, A., Gujar, A., Harrison, B., O'Hara, K. Sellen, A. A diarystudy of work-related reading design implications for digital reading
devices. CHI'98. p. 241 - 48.
Agarawala, A. Balakrishnan, R. Keepin' It Real: Pushing the
Desktop Metaphor with Physics, Piles and the Pen. CHI '06.
Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G., Buxton, W.
Digital Tape Drawing. UIST'99. p. 161-70.
Bondarenko, O., Janssen, R. Documents at Hand: Learning from
Paper to Improve Digital Technologies. CHI 2005.
Bragdon, A., Zeleznik, R., Williamson, B., Miller, T., LaViola, J.
GestureBar: improving the approachability of gesture-based
interfaces. CHI'09. p. 2269-78.
Brandl, P., Forlines, C., Wigdor, D., Haller, M., Shen, C. Combining
and measuring the benefits of bimanual pen and direct-touch
interaction on horizontal interfaces. AVI 2008, p. 154-61.
Brandl, P., Leitner, J., Seifried, T., Haller, M., Doray, B., To, P.
Occlusion-aware menu design for digital tabletops. CHI 2009
Extended Abstracts. p. 3223-28.

9.
10.
11.

13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.

39.

Buxton, B., Sketching User Experiences: Getting the Design Rightand the Right Design. 2007, San Francisco: Morgan Kaufman.
Buxton, W. Chunking and Phrasing and the Design of HumanComputer Dialogues. IFIP `86, p. 475-80.
Buxton, W., Myers, B. A Study in Two-Handed Input. CHI'86.
Chen, N., Guimbretiere, F., Dixon, M., Lewis, C., Agrawala, M.
Navigation Techniques for Dual-Display E-Book Readers. CHI '08.
Cohen, P. The role of natural language in a multimodal interface.
UIST'92. p. 143-49.
Cohen, P., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I.,
Chen,L., Clow,J. QuickSet: Multimodal Interaction for Distributed
Applications. ACM Multimedial 97.
Davidson, P.L., Han, J.Y. Extending 2D object arrangement withpressure-sensitive layering cues. UIST '08.
Engelhard, L. Native Dual Mode Digitizers: Supporting Pen, Touchand Multi-Touch Inputs in One Device on any LCD. Society for
Information Display SID 08 Digest. p. 1306-09.
Frisch, M., Heydekorn, J., Dachselt, R. Investigating Multi-Touchand Pen Gestures for Diagram Editing on Interactive Surfaces ITS'09 Conf. on Interactive Tabletops and Surfaces.
Greenberg, S., Buxton, B. Usability evaluation considered harmful(some of the time). CHI '08. p. 111-20.
Guiard, Y., Asymmetric Division of Labor in Human Skilled
Bimanual Action: The Kinematic Chain as a Model. The Journal ofMotor Behavior, 1987. 19(4): p. 486-517.
Hartmann, B. Pictionaire: Supporting Collaborative Design Work by
Integrating Physical and Digital. CSCW '10.
Hinckley, K., et al. Codex: a dual screen tablet computer. CHI '09.
Hinckley, K., Pahud, M., Buxton, B. Direct Display Interaction via
Simultaneous Pen+Multi-touch Input. SID 2010 Digest, p. 537-40.
Hinckley, K., Pierce, J., Sinclair, M., Horvitz, E. Sensing Techniquesfor Mobile Interaction. UIST 2000, p. 91-100.
Hinckley, K., Zhao, S., Sarin, R., Baudisch, P., Cutrell, E., Shilman,
M., Tan, D. InkSeine: In Situ Search for Active Note Taking. CHI'07.
Kidd, A. The marks are on the knowledge worker. CHI '94. 186-91.
Kurtenbach, G., Buxton, W. Issues in Combining Marking andDirect Manipulation Techniques. UIST'91.
Kurtenbach, G., Fitzmaurice, G., Baudel, T., Buxton, B. The Designof a GUI Paradigm based on Tablets, Two-hands, andTransparency. CHI'97, p. 35-42.
Li, Y., Hinckley, K., Guan, Z., Landay, J. Experimental Analysis ofMode Switching Techniques in Pen-based User Interfaces. CHI '05.
Marshall, C., Bly, S. Saving and Using Encountered
Information:Implications for Electronic Periodicals. CHI '05.
http://en.wikipedia.org/wiki/ Microsoft_Courier. 2009.
Roth, V., Turner, T. Bezel swipe: conflict-free scrolling and multiple
selection on mobile touch screen devices. CHI 2009.
Ruiz, J., Bunt, A., Lank, E. A model of non-preferred hand modeswitching. Graphics interface 2008 p. 49-56.
Ruiz, J., Lank, E. A study on the scalability of non-preferred handmode manipulation. ICMI '07, p. 170-77.
Schilit, B., Golovchinsky, G., Price, M. Beyond paper: supporting
active reading with free form digital ink annotations. CHI'98.
Sellen, A., Kurtenbach, G., Buxton, W., The Prevention of Mode
Errors through Sensory Feedback. J. Human Computer Interaction,
1992. 7(2): p. 141-64.
Terrenghi, L., Kirk, D., Sellen, A., Izadi, S. Affordances for
manipulation of physical versus digital media on interactive
surfaces. CHI '07. p. 1157-66.
Wilson, A.D., Izadi, S., Hilliges, O., Garcia-Mendoza, A., Kirk, D.
Bringing physics to the surface. UIST 2008.
Wu, M., Balakrishnan, R. Multi-finger and whole hand gesturalinteraction techniques for multi-user tabletop displays. UIST 2003.
Wu, M., Shen, C., Ryall, K., Forlines, C., Balakrishnan, R., Gesture
Registration, Relaxation, and Reuse for Multi-Point Direct-Touch
Surfaces, IEEE Intl. Workshop on Horizontal Interactive HumanComputer Systems. 2006.
Yee, K.-P. Two-Handed Interaction on a Tablet Display. CHI 2004.