Manual Deskterity : An Exploration of
Simultaneous Pen + Touch Direct Input
Ken Hinckley1

Jenny Rodenhouse2

Abstract

kenh@microsoft.com

jrodenh@microsoft.com

Koji Yatani1,3Andy Wilson1

koji@dgp.toronto.edu

awilson@microsoft.com

Michel Pahud1

Hrvoje Benko1

Manual Deskterity is a prototype digital drafting tablethat supports both pen and touch input. We explore adivision of labor between pen and touch that flows fromnatural human skill and differentiation of roles of thehands. We also explore the simultaneous use of pen
and touch to support novel compound gestures.

mpahud@microsoft.com

benko@microsoft.com

Nicole Coddington2

Bill Buxton1

nicolec@microsoft.com

bibuxton@microsoft.com

Keywords
Pen, touch, gestures, tabletop, tablets, bimanual input

ACM Classification Keywords
H.5.2 Information Interfaces and Presentation: Input

General Terms
Human Factors, Design

Introduction
1

Microsoft Research

32

University of Toronto

Microsoft Corporation

Dept. of Computer Science

One Microsoft Way

40 St. George St. Rm. 5170

Redmond, WA 98052 USA

Toronto, ON, M5S 2E4, CANADA

Copyright is held by the author/owner(s).
CHI 2010, April 10–15, 2010, Atlanta, Georgia, USA.
ACM 978-1-60558-930-5/10/04.

We are witnessing a shift towards systems employing
direct manual input where the user interacts directlywith the display, rather than indirectly, as with themouse and cursor of traditional GUI’s. This has
renewed interest in both pen and touch input, in form
factors ranging from hand-helds, slates, desk-tops,
table-tops, and wall displays. The iPhone, Tablet PC,
Wacom Cintiq, Microsoft Surface, and Smartboard are,respectively, examples of each. Neither multi-touch nor
pen input are new, but few systems explore their use inconjunction [3,27,28]. The presence of both modalities
may alter our perspective on multi-touch input; thesame can be said for pen gestures. Hence simultaneous
pen + touch is a nascent topic in need of further study.

This argues for a holistic approach rather than focusedevaluation of individual techniques [12], even though
this is often not rewarded by CHI. Our experience isthat trying many ideas [4]— some good, some bad,and some intentionally chosen to highlight conflicting
conventions or thorny design decisions rather than hidethem [11]— is an excellent way to draw out nuancesand gain insights into novel input modalities (e.g.
[17,20,21]). Our systems-oriented approach offers arealistic perspective of how combined pen and touch
input influences UI design issues and trade-offs.
The result is Manual Deskterity, a scrapbooking application inspired by how designers work with designboards and notebooks [4,14]—plus our experienceswith related prototypes [15,18]. We advocate a divisionof labor between pen and touch: the pen writes, touch
manipulates, and the combination of pen+touch yields
new tools. This articulates how our system interprets
unimodal pen, unimodal touch, and multimodal pen +touch inputs, respectively. We contribute novel pen +touch gestures, while also raising, by way of examples,design questions that probe how the roles of pen andtouch should be differentiated (or not) in UI design.

Related Workfigure 1. Manual Deskterity prototype:
The pen writes, touch manipulates,and pen+touch yields new tools.

Many current direct input systems employ only one oftouch or pen input. Yet an earlier generation of devices,such as the Bell-South/IBM Simon smartphone (1993),
the Psion Series 5 PDA (1995), and the Palm Pilot(1996), supported use of either pen or touch. Part ofwhat limited these earlier systems was that thetechnology could not differentiate pen contact fromfinger contact. Emerging dual-mode digitizers
distinguish pen and touch [9], but existing drivers do
not yet support the two simultaneously.

Several research efforts explore the combination of pen
and touch. Yee [28] uses single-touch + pen input tosupport panning a canvas while drawing with the pen.
Wu [27] describes two combined pen and touchgestures. Brandl [3] explores bimanual pen + multitouch techniques that assign the pen to the preferredhand and touch to the nonpreferred hand. We teaseapart the factors of (1) pen vs. touch, (2) preferred vs.
nonpreferred hand assignment, and (3) unimanual vs.
bimanual interaction. For example, we consider
unimanual cases where the user interleaves pen andtouch interactions with the preferred hand, and we
explore a wider vocabulary of novel pen + touchgestures that afford compound transactions.
Cohen discusses the complementary role of natural
language and pen gestures [7]; he treats multimodal
input with a probabilistic approach [8]. We instead treat
pen+ touch input in a manner that affords deterministic
state-machine-driven GUI’s [6]. Also, because pen andtouch are both manual input modalities, the nuances of
how the two complement one another are more subtle,
and we must overcome a longer legacy of designs thathave treated pen or touch interchangeably.
Guiard [13] observes that the hands cooperate toaccomplish tasks, so the question is not ―Which hand isbetter?‖ but rather ―What is the logic of the division oflabor between the hands?‖ Likewise, in our research we
ask: What is the logic of the division of labor between
pen and touch in interface design? Guiard observes
that the nonpreferred hand frames the action of thepreferred hand. Our bimanual pen+touch gestures buildon this: pen gestures act upon an object, and arephrased together by muscular tension [5] from theuser’s nonpreferred-hand fingers held on the object.

Frisch describes a user-elicited collection of touch andpen gestures [10], and reports that users often treat
pen and touch interchangeably. But as a result, theuser-defined gesture set contains many ambiguities.
Which pen, touch, or pen + touch gestures should asystem support (or not support), and why? In our
experience, user-defined gestures are insightful but
must be taken with a grain of salt because users havedifficulty envisioning how they would employ newmodalities of which they have little or no experience.
Indeed, our exploration of Manual Deskterity convinces
us that if each input modality offers complete coverageof all possible interactions, it quickly robs the
combination of pen and touch of much of its vigor.
Differentiating between pen and touch, rather thantreating them interchangeably, offers a consistent andrich designed input vocabulary. Nonetheless we build
our gestures on a vocabulary of natural occurring
bimanual actions, as shown in the following study.

Design Study Using a Paper Notebook

figure 2. User behaviors observed
during design study. (a) Users tuck the
pen between fingers while manipulating
items. (b) Thumb and forefinger grasp
an item while writing about it. (c) Cut
scraps fall onto the work surface. (d)Users often pull tools and new content
onto the notebook from above.

We conducted an observational study to gain insight
into how people naturally work with pens, tools, andpieces of paper. We asked each participant to illustrateideas for a hypothetical short film by pasting andannotating clippings in a paper notebook. To simulate aslate computer where the user could move between
pages, we provided a small paper notebook as theauthoring space. We provided users with pens, tape,scissors, and 20 sheets of inspirational materials.
Eight people participated in the study. We looked forpatterns in how users gestured and structured theirworking space. We observed behaviors (B1-B9) that
informed specific gestures and features in our system:

B1. Participants tucked the pen between the fingers of
their preferred hand when interleaving writing andmoving clippings (Fig. 2a). People were remarkably
adept at interleaving pen and touch in this manner.
As a result we consider unimanual multi-touchgestures performed while the pen is tucked.
B2. Participants temporarily held clippings in place with
one finger of the nonpreferred hand (Fig. 2a).
B3. Participants exhibited a strong tendency to hold aclipping with their nonpreferred hand while writing
about it with the pen (Fig. 2b).
B4. A common hand posture was to frame a clippingwith thumb and index finger (Fig. 2b) while writing
about it. This seemed to help users mentally focus
on a source object and reference annotations to it.
B5. Participants used only parts of the inspirational
materials. They cut a sheet while holding it in theirnonpreferred hand, above the notebook. The
unwanted part fell onto the page (Fig. 2c). Users
occasionally adopted these scraps into their work.
B6. Participants arranged the workspace with thenotebook proximal to their body, while reaching
above it to access tools and materials (Fig. 2d).
B7. Piling clippings was a common behavior. Users
formed piles of ―interesting‖ items while holding
the remaining items in the nonpreferred hand.
B8. While not a common behavior, a couple of people
did employ clippings as a constraint for the pen, todraw a border around an item (Fig. 3e).
B9. Tearing sheets of paper was a bimanual behavior
performed with the fingers (Fig. 3f).
The behavioral observations above form a valuable
contribution. By starting from a suitable task context
with physical objects, our approach elicits a naturallyoccurring set of unimanual and bimanual behaviors
from users, with both pen and bare-hand manipulation,that exhibits a clear differentiation between the roles of
pen and touch. Terrenghi’s design study notes a similarfinding: physical manipulation of real-world clippings ona tabletop yielded rich bimanual interactions, whereas
corresponding digital ones did not [25]. These stand incontrast to the results of user-elicited gestures [10],which might otherwise lead us to treat pen and touch
interchangeably. We feel it would be mistaken to do so.

figure 3. Additional Behaviors
observed during design study.
(e) Drawing a border around a clippingby holding it with the nonpreferredhand while using its edge to constrain
the path followed by the pen held in
the preferred hand. (f) Tearing a pageby anchoring it with a thumb whilepulling it away using fingers of the
opposite hand. Note that in thisexample, the pen is again tucked
between the fingers of the preferredhand, demonstrating how prevalent
this was with our participants during avariety of manual activities.

We should also emphasize here that our intent is not tomimic the specific actions required to work withphysical paper. Pen + touch gestures should go beyond
physical paper, but the best foundation for suchgestures likely lies in behaviors that people alreadyexhibit when working with pen, paper, and clippings.

Implementation
We use Microsoft Surface for our prototype. The pen
uses an infrared LED, activated during surface contact
via a tip switch. The pen is much brighter thanhand/finger contacts, so we can robustly identify the
pen as the brightest spot in the image. The software is
written in C# with WPF and the Microsoft Surface SDK.
A potential limitation of pen + touch input is the socalled ―palm rejection‖ problem: the user may rest his
hand on the screen while writing, potentially leading tounintended operations. Brandl [3] makes no mention ofthis issue, even though he shows a black glove on theuser’s pen hand to prevent the digitizer from reacting
to palm contact. We treat touches with a large contact
area as incidental, which is sufficient for prototyping
pen + touch techniques. Robust handling of incidental
contacts remains an important problem for future work.

Application Scope and Motivation
Manual Deskterity is in some respects a ―toy‖ system
intended primarily as a research vehicle to explore pen+ touch, which we believe has many potential
applications. Nonetheless we emphasize practical
functionality well-suited to ―idea collection,‖ notetaking, and mark-up functionalities that have been well
documented by previous research (e.g. [15,18,23]).
For example, several papers note the importance ofwriting, annotation, selecting, copying, arranging, andaggregating objects both in digital [1,27] and physicalcontexts [2,19,26]. Our design study suggests
additional behaviors of interest in the context of pen +touch, such as holding items while acting on them withthe pen (B2, B3, B4), cutting and tearing operations(B5, B9), creating objects from the space above thenotebook (B6), and employing clippings as a constraint
for the pen (observation B8). These formed the basis of
the features we elected to explore using pen, touch,and pen+touch interactions.

Core Tasks: Pen Writes, Touch Manipulates
The core interactions of Manual Deskterity are driven
by multi-touch interactions including zooming, flipping
pages, moving objects, selecting objects, and creating
new objects such as digital post-it notes. The othercentral task is writing; here, only the pen produces ink
strokes (although in some contexts the finger ―smears‖
colors, as discussed later). Notwithstanding theexception of finger-painting, for these core tasks thepen writes, and touch manipulates, period. This makes
the entire canvas, and objects on the canvas, available
for immediate annotation with the pen, while
pan/zoom, page navigation, and object manipulation
are also immediately available via touch, without any
explicit mode switches.

Selecting and Manipulating Objects
Our prototype scrapbooking application includes objects
such as photos and post-it notes that the user arranges
on the canvas via direct manipulation (single touchdrags, while multi-touch rotates and scales). The user
defines a collective scope (i.e., multiple object
selection) via finger-taps that incrementally add
individual objects to the current selection. Dragging an
item that is already selected drags it and all otherselected items while maintaining their relative spatial
relationship. Dragging an item that is not alreadyselected drags only that item; this enables multi-touchdragging of items to multiple different locations.

figure 4. Selection and Context Menu.
(Top) Selection feedback consists of apink highlight and a large drop shadow
to visually ―float‖ the object.
Deselected objects have only a thin
drop-shadow. (Bottom) Context
menus use a radial (marking) menu.
The user touches pen or finger to thered dot and strokes towards thedesired command. This represents one
situation where it is necessary tosupport both pen and touch
interchangeably to match user
expectations, even though elsewhere
the system sticks to a differentiation ofroles between pen and touch.

Context menus. On selection, a radial menu appears at
the upper right corner of objects (Fig. 4). Initially, ourradial menus required use of the pen, but users
uniformly expect this menu to be operable via touch aswell. As a high level principle we still advocate
differentiation of roles: the pen writes, and touch
manipulates. But acting on the radial menu represents
a limited spatially-multiplexed context where pen andtouch should indeed be treated interchangeably.
Creating new objects. In our design study we observedthat users bring in new materials from above (B6). We
incorporate this observation into our interface design
via a finger-activated bezel menu that builds on Bezel
Swipe [24]. The user performs a continuous finger-dragthat crosses the top screen bezel and onto the canvas
to create and position objects of various types, such as
digital post-its (see video). The user can then annotate
the new object immediately.
Summary. By supporting core manipulative tasks withtouch, and with touch only, Manual Deskterity enables

users to fluidly interleave annotation and othersecondary tasks. This approach also supports gracefuldegradation to one-handed usage, which we believe isan important property to afford mobile pen+touchform-factors when they become available in the future.
Nonetheless two-handed interaction is encouraged
when the usage context makes it suitable (e.g. working
at a desk, rather than while mobile). The key is thatthe mode switch is in the user’s hand: he can work
one-handed and flip between pen and touch by rapidly
tucking the pen between fingers, or he can work with
two hands by performing most touch operations with
the nonpreferred hand, while writing and annotatingwith the pen in the preferred hand.

Pen + Touch Yields New Tools
Next, we explored how the expanded input vocabulary
afforded by the combination of pen and touch can give
new tools. Our design keeps the primitives forpen+touch operations simple. The richness of thegestures arises from how the primitives are combined.
For the pen the primitives we use are tap, drag-off,crossing, or drawing a stroke. For multi-touch, we
employ single-finger tap, single-finger hold (i.e. a tap
with a long duration, as seen in our design study,
behavior B2), holding with thumb and forefinger (B4),and crossing. We do not implement all combinations,but rather support a sufficiently rich set of operations,with semantics that map well to our application domain,
to illustrate the expressiveness of our approach.
The pen+touch techniques described below all use
fingers of the nonpreferred hand hold an item, while
the pen acts in reference to the item. This builds on thetendency that we observed for users to hold clippings
with the nonpreferred hand while making pen markings

in reference to them (B3), and it also corresponds wellto Guiard’s principle that the nonpreferred hand sets
the frame of reference for the preferred hand [13].
Thus, we construct gestures that allow for non-physicaldigital effects, yet remain grounded in people’snaturally occurring behaviors with physical paper.

figure 5. Stacks of items formed via
the Stapler pen+touch gesture. The
user staples items by (1) tap-selecting
a series of items, (2) holding a finger
on the representative item for the
stack, and then (3) tapping the pen onthat item while continuing to hold it.
The representative item appears on the
top of the resulting stack.

figure 6. Intricate X-Acto cuts formedby one user during an informal
evaluation of our system.

Stapler: Grouping Items into a Stack
To support piling (design study B7) in an intuitive waythat reduces the manual effort required to drag widely
scattered items into piles, we support stapling items
into a stack (Fig. 5). The user can finger-tap-select anumber of items, and then staple all of them together
by holding an item and tapping it with the pen. The
item that the user holds appears on the top of thestack, thus promoting it to represent the entire stack.
Tap-selecting a series of items and stapling them
together enables quick tidying of a messy work surface
into a few piles (see video). This transaction separates
the identification of the items to stack from the decisionof which item should become the representative itemon the top of the resulting stack. Performing the
pen+touch gesture on the representative item keeps
the user’s attention focused on it, at exactly the
moment the user makes this decision, which seems tocorrespond well with users’ mental model of the task.
X-acto Knife: Cutting Items (and Tearing Items)
The user can turn the pen into an X-acto knife byholding an object and fully crossing it with the pen.
That is, the pen stroke starts outside the object,crosses through the interior of the object, and finishes
on the exterior of the object. The pen stroke within theitem can follow any path, allowing intricate cuts if
desired (Fig. 6). When the pen exits the item, both the

cut and the scrap piece appear on the page, following
the real-world action where we observed scraps falling
onto the work surface (design study B5).
To probe the semantics of pen vs. touch in analogous
gestures, we also implemented tearing items by holding
an object with a single finger, and then crossing theitem with another finger (study, B9). This tears theitem along the line connecting the entry and exit pointsof the finger. The operation is similar to cutting, but
produces a different visual affordance in the scraps.
This technique demonstrates how touch can sometimes
be used to sneak a nuance of expression into atransaction, by applying a different look or different
default command parameters. On the other hand, in
our system this precludes using the touch gesture for adifferent command, such as layering. Should thesemantics of analogous pen+touch vs. touch+touchgestures be similar or contrasting? This remains an
open design question raised by this example.
Carbon Copy: Drag-Off with the Pen
The user copies an object by holding with a finger (B2)
and then ―peeling off‖ a copy with the pen (Fig. 7). This
gesture is similar to a copy gesture identified by Frisch[10], but here we identify the interaction pragmatics aswell as why this gesture differs from a Copy command.
While we have so far been somewhat critical of theuser-elicited gestures methodology, in this case we
seen an example of how the methodology can yieldfertile ground for suggesting plausible gestures (so longas we keep in mind that users are not designers).
Once the pen drags away by a minimum distance, asemi-transparent copy of the object appears attachedto the pen. As the pen continues to drag, the object

becomes opaque. The user may then proceed to dragthe object to ―paste‖ it at the desired location. Anyannotations on top of the object are also copied.

figure 7. Duplicating an item byholding it with a finger of the nonpreferred hand while dragging awaywith the pen.

The properties of this approach differ from using the
Copy command of the object’s context menu (Fig. 4,bottom). Drag-off with the pen phrases together theentire transaction (select, copy, and drag to final
position) into a single cognitive chunk [5] via themuscular tension of the nonpreferred hand holding
down the original item. We found that this corresponds
well to users’ mental model of duplicating items– they
not only want to copy the item, but also place theduplicate at a particular location. By contrast, Copy
from the context menu divides the select-copy-positiontransaction into multiple steps, enabling one-handed
copying at the cost of more syntactical complexity [5].
Holding an item and peeling off a copy with the pen is agood example of a pen+touch technique that ostensibly
violates the principle that the pen writes and touch
manipulates because here the pen drags the copy.
However, those principles apply to pen or touch as
unimodal inputs. The transaction is consistent with theprinciple that we use to guide multimodal gestures: pen+ touch yields new tools. The gesture feels natural andeffective because it is grounded in people’s naturally
occurring behaviors with physical paper, such as
holding an item and making pen strokes in reference toit (Guiard [13], and design study B3).

figure 8. Composing the Ruler toolwith the X-acto tool. The user can cut
along a straightedge, including aroundcorners, as shown here, to produce
interesting cutting effects.

Ruler: Using an Object as a Straightedge
The user can employ an object as a straightedge byholding down the object with the thumb and index
finger, like the framing gesture observed in our design
study (B4, Fig. 2b). The user can then stroke along theobject with the pen stroke constrained to its border, as
inspired by observation B8.
When two fingers come down on an object, ananimation starts that increases the transparency of theitem and adds a dotted line around its border. Informal
test users who tried early versions of the systemsuggested that items should become mostlytransparent in this manner so that users can see therelationship of the straightedge to the underlying
surround of other objects and strokes.
The Ruler uses content as its own tool, introducing asubtle duality between content and tool into oursystem. We could add a dedicated ruler object, but thiswould necessitate acquiring the ruler before drawing astraightedge. Using an object as its own straightedge
also supports other sketching techniques, such asadding a drop shadow or ―outer glow‖ to an item. To beclear, however, we are not arguing that the systemshould not include a dedicated ruler tool; rather, we arearguing that using an object as its own straightedge
has some interesting interaction design properties thata dedicated ruler tool does not.
Composition of the Straightedge with Cutting
To illustrate how our interaction design allows multiple
atomic interactions to be composed together intohigher-level input phrases, we implemented X-acto
cutting along straightedges (Fig. 8). To do this, theuser first finger-tap-selects the photo to cut, and thenuses the two-finger thumb and index finger grasp on an
overlapping item to establish a straightedge. The user
can then stroke the pen across the selected photo
along the straightedge to cut it in a straight line. The
user can even cut around the corner of the masking

figure 9. (top) Calling up additionalcommands via the Finger Shadow.
(bottom) Finger painting from thecolor pots. Each successive finger paintstroke is progressively fainter, until ―allthe ink is rubbed off‖ of the user’s
finger. In addition to mimicking
physical media such as charcoal
sketching, this facilitates a lightweighttransition out of the finger-paintingmode.

object (Fig. 7). This composition of techniques ispossible because tapping provides a way to define a―collective‖ scope of more than one object, and because
holding phrases together multiple pen and touch inputs
into a single ―chunk‖ that the system can interpret as aunitary command [5].

or move it (or touching other controls, such as thebezel menu, or flipping pages) always takes precedence
over finger painting. Otherwise, a more explicit means
to ―stop‖ finger painting and return to the default
behavior that touch manipulates would have to be
introduced, and we did not wish to do so at this point.

Brush & Stamp: Using an Object as Its Own Tool
Our system includes a couple of techniques, which wecall stamping and brushing, for producing creative
effects by letting the user employ content on the page
as its own tool. Due to space constraints, these areillustrated only in the accompanying video figure.

Summary and Discussion
Let us reflect briefly on how we use touch to holditems, in combination with one or more pen strokes
that act in reference to the item that the user isholding. We have seen that this approach enforces astrong notion of phrasing [5] in the resulting interface
design: there are no persistent modes, but rather tool
use is always tied to holding an object with thenonpreferred hand. There is no possibility of getting
stuck in a tool mode with the pen, nor is there ever aquestion of how to switch back to the default action ofdrawing ink strokes on the page. Once the user
releases objects, drawing on the page with the pen
always leaves ink strokes.

Finger Shadow & Finger Painting
We have also experimented with the Finger Shadow(Fig. 9, top), which uses a finger-tap on the canvas tobring up in-place commands (see video figure). This
accesses additional commands, including color pots.
In early demonstrations of our system we observedthat most people expect touching the color pots toenable smearing of colors with the finger. To probe thisissue further, in this context we intentionally break ourdesign rule for unimodal inputs that the pen writes andtouch manipulates. To prevent finger painting frombecoming a heavyweight mode, each successive finger
paint stroke appears fainter until ―all the ink has been
rubbed off‖ the user’s finger (Fig. 9, bottom). This also
mimics how artists naturally work with some physicalmedia, such as charcoal sketching.
With finger painting, we have to face a genuine design
dilemma: when ink remains on one’s finger, should it
be possible to finger paint on top of objects? In oursystem, the answer is no. Touching an object to selectThis approach is akin to nonpreferred-hand modeswitching [16,22], but requires no physical button andthus readily scales to a plurality of modes [25]. It alsooffers an additional advantage: object selection is
integrated with the mode switch itself when the user’s
hand touches down on an object on the display. Hencea unique design property of pen+touch is the facility
with which it can support modes and tools specific toparticular objects on the screen.

Informal Usability Evaluation
In addition to our design study, and observations frominformal demonstrations, we also conducted a usability
evaluation with seven professional designers. Due tospace constraints we note only a few observations
here. Users found the core operations of writing andmanipulating objects via touch to be largely obvious.
Dragging new items from the top bezel was particularly
well received. The general pattern of our combined pen+ touch gestures resonated with users, but users did
have to be told how to articulate the gestures when
first encountering them. After the study, one designercommented that ―the way it works is just like the way I
already work in my notebook!‖ Another usercommented that ―I wouldn’t have guessed the gestures
work that way, but once I tried it, it felt pretty natural.‖Conclusion and Future Work
Our work is motivated by a desire to extend use of pen
and touch, including their use in concert, in order toenable users to take better advantage of each. Inherent
in this statement is our belief that each has its own
strengths and weaknesses, and that these are largely
complementary. Yet, what should be the division oflabor between pen and touch input when both areavailable? What is the design vocabulary afforded by
combining pen and touch? What are some potential
limitations? Such questions provide a sense of thespace that we set out to explore in this study. Our
purpose has not been to advocate any particular
design; rather, to conduct a quick informal probe thatyields insights into how interface designs can effectively
combine pen and touch input.
The fluency, flow and engagement afforded by oursystem (despite the relatively primitive and sometimes
arbitrary tools tested), is an encouraging existence
proof that the path which we set out on is worth
probing more deeply and systematically. While we do
not claim to have created a revolutionary new graphic

design package, our system offers many examples andinsights that should be of considerable interest to theother researchers and designers who are now actively
exploring and seeking to characterize the design spaceof pen+touch techniques.
We have observed that users, even with limited
experience, are able to engage with the system. This
demonstrates that pen + touch input on a direct display
can afford a design where the pen writes, touch
manipulates, and the combination of pen and touchyields new tools. A number of simple pen or touchactions (holding, tapping, dragging, crossing) can becomposed in a phrase structure, delineated by themuscular tension of holding an object or state, thataffords a rich design space of interactions. All this canbe done without using a button on the pen, keys for
non-preferred hand mode switching [16,22], or
permanently visible UI that detracts from the user’s
focus on the workspace [20].
We have seen that if pen+touch yields new tools,
implicitly this means that in some contexts we will
violate the first two principles: the pen does not always
write, nor does touch always manipulate. Our
explorations have convinced us that if a system strictly
limits itself so that the pen ONLY writes, and touch
ONLY manipulates, this leads to a simple and consistent
but artificially crippled system. Hence the design tradeoffs we explore are intricate for good reason.
Furthermore, we could have presented one or two of
the techniques in this paper, with formal studies, butchose not to do so because in our view the primarysignificance of pen + touch accrues from the richdesign space of compound gestures that is afforded,
rather than from the individual, atomic techniquesthemselves. For example, for any one of oursimultaneous pen+touch techniques, one could likely
devise a pure multi-touch technique that works nearly
as well. But we believe it would be difficult to design asystem supporting the same breadth of operations
without resorting to increased modes, on-screen
widgets, and a larger set of arbitrary and/or more
complex touch gestures.
In addition to proving that our approach scales from ademo application to a full-blown one, it also remains todemonstrate how these techniques serve to enhancethe effectiveness and user experience of otherapplications, such as working with a spread-sheet, forexample. We believe that they can and will, but that isa long-shot from actually doing so. As well, despite ourhopes for these techniques on other form factors, thefact remains that we have not yet done those tests.
There is still work to do.

References
[1]
[2]
[3]

[4]
[5]
[6]
[7]
[8]
[9]

Agarawala, A., Balakrishnan, R. Keepin' It Real: Pushing the
Desktop Metaphor with Physics, Piles and the Pen. CHI 2006.
Bondarenko, O. and Janssen, R. Documents at Hand: Learningfrom Paper to Improve Digital Technologies. CHI 2005.
Brandl, P., Forlines, C., Wigdor, D., Haller, M. and Shen, C.
Combining and measuring the benefits of bimanual pen anddirect-touch interaction on horizontal interfaces. AVI '08 Conf.
on Advanced Visual interfaces. p. 154-61.
Buxton, B., Sketching User Experiences: Getting the Design
Right and the Right Design. 2007, Morgan Kaufman.
Buxton, W. Chunking and Phrasing and the Design of HumanComputer Dialogues. IFIP Information Processing `86.
Buxton, W. A three-state model of graphical input. Proc.
INTERACT'90. p. 449-56.
Cohen, P. The role of natural language in a multimodal
interface. UIST'92. p. 143-49.
Cohen, P., et al. QuickSet: Multimodal Interaction forDistributed Applications. ACM Multimedial 97.
Engelhard, L. Native Dual Mode Digitizers: Supporting Pen,
Touch and Multi-Touch Inputs in One Device on any LCD.
Society for Information Display SID 08 Digest.

[10] Frisch, M., Heydekorn, J. and Dachselt, R. Investigating MultiTouch and Pen Gestures for Diagram Editing on Interactive
Surfaces. ITS '09 Conf. on Interactive Tabletops and Surfaces.
[11] Greenberg, S., Boyle, M. and LaBerge, J., PDAs and Shared
Public Displays: Making Personal Information Public, andPublic Information Personal. Personal Technologies, 1999.
3(1): p. 54-64.
[12] Greenberg, S. and Buxton, B. Usability evaluation consideredharmful (some of the time). CHI '08. p. 111-20.
[13] Guiard, Y., Asymmetric Division of Labor in Human Skilled
Bimanual Action: The Kinematic Chain as a Model. The Journalof Motor Behavior, 1987. 19(4): p. 486-517.
[14] Hartmann, B. Pictionaire: Supporting Collaborative Design
Work by Integrating Physical and Digital Artifacts. CSCW
2010.
[15] Hinckley, K., et al. Codex: a dual screen tablet computer. CHI'09. p. 1933-42.
[16] Hinckley, K., Guimbretiere, F., Baudisch, P., Sarin, R.,
Agrawala, M. and Cutrell, E. The Springboard: Multiple Modesin One Spring-Loaded Control. CHI 2006.
[17] Hinckley, K., Pierce, J., Sinclair, M. and Horvitz, E. Sensing
Techniques for Mobile Interaction. UIST 2000. p. 91-100.
[18] Hinckley, K., et al. InkSeine: In Situ Search for Active Note
Taking. CHI 2007. p. 251-60.
[19] Kidd, A. The marks are on the knowledge worker. CHI '94.
[20] Kurtenbach, G. and Buxton, W. Issues in Combining Marking
and Direct Manipulation Techniques. UIST'91. p. 137-44.
[21] Kurtenbach, G., Fitzmaurice, G., Baudel, T. and Buxton, B.
The Design of a GUI Paradigm based on Tablets, Two-hands,
and Transparency. CHI'97, p. 35-42.
[22] Li, Y., Hinckley, K., Guan, Z., Landay, J., Experimental
Analysis of Mode Switching Techniques in Pen-based User
Interfaces. CHI 2005. p. 461-70.
[23] Marshall, C. and Bly, S. Saving and Using Encountered
Information: Implications for Electronic Periodicals. CHI 2005.
[24] Roth, V. and Turner, T. Bezel swipe: conflict-free scrolling andmultiple selection on mobile touch screen devices. CHI 2009.
[25] Ruiz, J. and Lank, E. A study on the scalability of nonpreferred hand mode manipulation. ICMI '07 Conf. on
Multimodal interfaces. p. 170-77.
[26] Terrenghi, L., Kirk, D., Sellen, A. and Izadi, S. Affordances for
manipulation of physical versus digital media on interactive
surfaces. CHI '07. p. 1157-66.
[27] Wu, M., et al., Gesture Registration, Relaxation, and Reuse forMulti-Point Direct-Touch Surfaces, IEEE Intl Workshop on
Horizontal Interactive Human-Computer Systems. 2006.
[28] Yee, K.-P. Two-Handed Interaction on a Tablet Display. CHI
2004.