Enhancing Naturalness of Pen-and-Tablet Drawing
through Context Sensing
Minghui Sun1,2, Xiang Cao1, Hyunyoung Song3,4, Shahram Izadi1, Hrvoje Benko1,
Francois Guimbretiere4, Xiangshi Ren2, Ken Hinckley1
12
3
4
Microsoft Research
Kochi Univ. of Technology,
University ofCornell University
Cambridge, UK; Beijing,
Kochi, Japan
MarylandIthaca, NY
China; Redmond, WA
smh@jlu.edu.cn,
College Park, MD francois@cs.cornell.eren.xiangshi@kochidu{xiangc, shahrami, benko,
hsong@cs.umd.edu
tech.ac.jpkenh}@microsoft.com
ABSTRACTcreative tasks such as drawing.

Among artists and designers, the pen-and-tablet combination is widely used for creating digital drawings, as digitalpens outperform other input devices in replicating the experience of physical drawing tools. In this paper, we explore
how contextual information such as the relationship between the hand, the pen, and the tablet can be leveraged in
the digital drawing experience to further enhance its naturalness. By embedding sensors in the pen and the tablet tosense and interpret these contexts, we demonstrate how
several physical drawing practices can be reflected andassisted in digital interaction scenarios.

Our approach to address this problem is by closely
examining the physical practices of drawing and translatingthem to the digital pen-and-tablet drawing experience toenhance naturalness. When we include human practices andintentions into the ecosystem, there are several factors
providing unique contexts and affordances that suit theparticular drawing at hand, such as the way the drawing
implement is grasped, the setup of the drawing surface, andthe spatial relationships between the drawing implement,
the surface, and the subject being drawn. There has notbeen work that specifically leverages such rich contextual
information beyond the pen itself to facilitate the task of
digital drawing. We aim to explore this design space in thispaper.

ACM Classification: H5.2 [Information interfaces andpresentation]: User Interfaces. - Graphical user interfaces.
General terms: Design, Human Factors

RELATED WORK

Keywords: Drawing, pen interface, context sensing.

As additional controls on a digital pen are limited,dimensions such as rolling [2], tilting [12], pressure [9],and finger gestures on the barrel [10] have been proposed
as alternative input methods. Most of them are used fortriggering abstract commands. In this paper, however, wetake a different approach and interpret physical contexts tofacilitate interactions in ways that directly analogize
existing drawing practices.

INTRODUCTION

The digital pen-and-tablet has become widely adopted byartists and designers, is used as a natural device for digitaldrawing. The pen-and-tablet takes on the metaphor of aphysical pen and paper, while the digital application
extends its functionality. For example, one digital pen can
mimic the visual effects of many drawing implements suchas a paintbrush or a sketching pencil. The tablet can beswitched between various canvases and layers. While suchdiverse virtual functionalities are appreciated by users,
most of them need to be explicitly manipulated using
abstract operations such as a mode switch, menu selection,
parameter adjustment, and so on. This arguably makes the
pen-and-tablet experience depart from the physical practiceof drawing and more into the purely digital realm. The
cognitive loads for users to understand, memorize, andrecall these abstract mappings may distract their attention
from the task itself. This can be particularly problematic for

There have been many digital creation tools that mimic
practices using physical implements and props. To name afew, the digital tape drawing [1] system retains thefundamental properties of the physical tape drawing
commonly used in an automobile industry and enables thisin the digital world. HandSCAPE [8] is an orientationaware digital tape measure that uses embedded sensors to
capture the linear measurements of physical workplaces. In
the pen space, Project Gustav [3] attempts to mimic thepaintbrush experience by constructing virtual brush strokes
in 3D using pen orientation information and physics
simulation. This paper differs from these works in that we
aim to understand the higher-level contexts of drawing
determined by the user, instead of reproducing lower-level
physicality.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee.
ITS 2011, November 13-16, Kobe, Japan.
Copyright 2011 ACM 978-1-4503-0871-7/11/11....$10.00.

Context sensing and its applications have been a popular
topic for several years. Hinckley et al. [6] demonstrate on amobile device that a proximity sensor, a capacitive touch
strip, and an orientation sensor can be combined to1

implicitly change the state of mobile applications. Codex[5] demonstrates how the configuration of a dual-display
tablet can be used to seamlessly switch between individual
work, usage as an ambient display, or collaboration withanother user. Comparatively, we are the first to explore
context sensing in a drawing application.

the computer via USB. Similarly, we attached an IMU to a
12-inch Wacom Cintiq 12WX tablet to measure its tilt
angle as well as orientation (rotation within the display
plane). This lightweight tablet can be easily tilted andreoriented while on a stand or held in hand.

In particular, researchers have attempted to understand thecontext of a device usage by capturing ways that people
hold the device [7, 10, 11, 13]. Graspables [11]demonstrates how an array of capacitive sensors can bewrapped around various form factors, such as a bar of soap
or a sphere, to allow different grips to be recognized andutilized for input. Kim et al. [7] and Wimmer et al. [13]both propose interaction techniques to automatically initiate
an application by observing the hand grip and theorientation of a mobile phone. Most relevantly, Song et al.
[10] presented the multi-touch pen, which supported
switching between three different digital pen effects using
different pen grips, in addition to other more abstract finger
gestures.

We now detail our interaction designs, categorized by thedimension of the contextual information as well as theirinterpretations. The key rationale here is to look at therelationship between the user, the pen, and the tablet as aholistic ecosystem, instead of focusing on a single device.

INTERPRETING CONTEXTS FOR INTERACTIONS

touch input image

capacitive sensor

roll
yaw
pitch

orientation sensor(IMU)

MOTIVATIONIMU

To further ground our research, we interviewed aprofessional oil painter with over 20 years of experience, as
well as observed her drawing a portrait. She owns over 50
types of oil brushes in her studio, and usually uses around10 different brushes in a single painting. In addition, sheuses spatulas to apply or scrape paints, and charcoal sticks
to sketch outlines. She holds the oil brush, the spatula, andthe charcoal stick with very different grips as well as atdifferent angles. Within a single painting, she needs tochange between tools at a high frequency to achievedifferent effects. When she does oil painting, she places her
painting canvas on a desktop easel, which offers four
different tilt angles of 25°, 45°, 65°, and 85°. She changes
different canvas tilt angles based on different drawing tasks.

tiltorientation

Figure 1. Pen and tablet augmented with sensors.

Artists hold different drawing implements with different
grips, often enforced through their professional training.
With our prototype, users can simply change the grip on the
pen to activate different virtual implements that will bementioned below. The grip is recognized by the system
combining input from both the touch sensor and theorientation sensor (IMU).

When the artist sketches still life, getting the correct
proportion is one of her main concerns. In order to do so,
she often closes one eye and holds up the charcoal/pencil infront of the eye to measure the physical subject being
drawn, and using a finger to mark the length. This is a very
common technique among artists.

Although [10] has briefly demonstrated using pen grips toswitch between three pen effects as one of the possible
capabilities of the multi-touch pen device, this work
fundamentally differs from it in that we are taking a
scenario-driven approach, directly aiming at facilitating theactivity of digital drawing as a whole, and does notnecessarily dictate a specific technology. And while thefocus of the vast majority of the interaction techniques in[10] is in improving the digital pen experience, we chose tosolely focus on implicit interpretations of contexts that
already exist in physical drawing practice. Moreover, theaddition of the orientation sensor both enables many new
virtual implements and interactions to be detailed later andsignificantly improves the grip recognition capability
compared to [10]. The pitch (i.e. tilt) angle of the pen helps
further differentiate grips that are otherwise similar. The
roll angle of the pen is used to normalize the touch input
image (circular shift along the barrel circumference), sothat the recognition is invariant to pen rotation along its
longitudinal axis. On the recognition algorithm side, weused a k-Nearest Neighbor (KNN) algorithm that matchesboth the touch input image and the pen tilt angle to

These interviews and observations verified the importanceand richness of contextual information in the drawing
practice, such as the grip on the implements and theconfiguration of the drawing surface. We created a
prototype to explore how we may sense and interpret suchcontexts to facilitate interactions in digital drawing.
PROTOTYPE PLATFORM

The multi-touch pen hardware [10], which is a Wacom
Cintiq grip pen enclosed in a cylindrical layer of capacitive
multi-touch sensor array to detect the finger/hand contact
image (20x10 px, gray scale) on its barrel. In addition, we
attached an orientation sensor (Inertial Measurement Unit
or IMU, model CHR-6dm AHRS) to the pen to sense its
full 3D orientation angles (yaw, pitch, and roll) (Figure 1).
Both the capacitive sensor and the IMU are connected to2prerecorded templates from the same user. Both the input
and the template image are Gaussian-blurred before thematching, in order to account for slight misalignment in theuser’s finger positions. Compared to attempting to segment
the fingers from the touch image as rotationally invariant
features as in [10], here template matching directly on the
raw touch image (normalized by the IMU) allows us tocapture subtle differences in grips, and is not susceptible tosegmentation errors. An empirically determined thresholdis set on the maximal acceptable distance between the input
and the nearest training sample to filter out non-gestures.
Six subjects participated in the recognition experiment toevaluate the classifier by using 4-fold cross-validation. We
were able to achieve 86.3% recognition rate for a set of 10
different pen grips with user-specific training data.

switched and operated with a single hand. Of special note is
the airbrush (Figure 2e), where fine movement of the index
finger emulates the air trigger found on physical airbrushes.
The “air pressure” is mapped to the distance that the finger
traveled backwards from the starting position. The further
the user moves their finger back, the higher the air pressure
is, and hence the denser the resulting paint.

In our demo application, the name of the currently
recognized drawing implement is displayed at the upper
right corner of the screen (Figure 2). In addition toswitching implements with static grips, where applicable
the user can also use finer finger movements to operate thecurrently selected implement, such as controlling anairbrush (detailed later), again directly analogizing the way
the respective physical implement is operated. To avoid
finger movement accidentally causing confusions to thegrip recognition, once the expected movement starts, thecurrent implement selection is locked, until a significant
change of the pen tilt that reactivates the grip recognition.

Proximity between Pen and Tablet : Alternative Usageof the same Implement

In the real world, the same physical implement has been
given different implicit operation modes. For instance, apencil can be used to sketch or to measure, depending onwhether it is used on the paper or in the air. This concept is
applied to our sketching pencil grip (Figure 2c). When the
pen is away from the tablet (out of tracking range), thesystem infers that the user is trying to use the pencil tomeasure, and displays a one-dimensional grid on the tablet
to facilitate the measuring. The angle of the grid rotates tomatch the current tilt angle of the pen, so that the user canmeasure the digital drawing at the same time and along thesame direction as she measures the physical object, thus
helping the user translate the proportion of the object to her
drawing.

(b)

(a)

(d)

(c)

Implements triggered and operated with two hands further
allow richer dynamic operations. The pencil sharpener istriggered by holding the two ends of the pen using both
hands, similar to holding a physical pencil inserted into a
sharpener (Figure 2g). A twisting action from either or both
hands operates it, resulting in adjusting the stroke width of
the sketching pencil implement, and then the user cancontinue sketching with the selected width.

(f)Tablet Tilt : Different Drawing Layers

People often configure their drawing surfaces at different
tilt angles for different types of drawings. For example,
when painting an outdoor scene, a near vertical canvas is
optimal to align the perspective of the drawing to thephysical surrounding; a horizontal desk suits cartoon
illustrations, especially when the artist needs to tracedrawings to create animations; for engineers, a slightly
slanted drawing board makes drawing with tools most
comfortable. The tilt angle of the drawing surface is a good
indication of the type of drawing being done.

(e)

(g)

Figure 2. Virtual implements. (a) Pen (b) Eraser (c)Sketching pencil (d) Chinese calligraphy brush (e) Airbrush(f) Stamp (g) Pencil sharpener for width adjustment.

In our prototype, we adopt the general notion ofinterpreting different tablet tilts as different drawing types.
Within the range of 0-90°, the user can tilt the tablet freely
to activate one of several drawing layers (3 in the current
implementation: 0-30°, 30-45°, 45-90°). In addition to theregular drawing layers, when the tablet is tilted towards theuser (120-180°), a private layer is activated where the user
can write private notes that are only visible in thisconfiguration. The note is conveniently hidden from otherpeople nearby, occluded by the tablet itself. Again this isconsistent with how people would hide information in thephysical world.

INTERACTION TECHNIQUESOur prototype currently supports the following virtual
implements that reflect physical grips and the orientation of
the tool.
In addition, the configuration of the tablet and its
relationship with the pen is also accounted for in our laterinteraction techniques.
Virtual instruments

Pen, Eraser, Sketching pencil, Chinese calligraphy brush,
Airbrush, and Stamp (Figure 2a-f) are examples of onehanded implements. These implements can be easily

Tablet Orientation : Different Subjects of Drawing

For both physical and digital drawing, artists usuallychoose canvases of various aspect ratios to suit different
3

types of drawing [4]. For example, today we used words
like portrait and landscape as generic terms to describe
aspect ratios, which precisely originated from the fact thatvertically elongated canvases are suited for portraying
people, and horizontally elongated canvases are suited for
depicting natural landscape.

leverages the benefit of the digital drawing to further aid
the drawing activity. Although we do acknowledge that apractical system may need to combine or compromise
between physical and abstract interactions, in this paper wedeliberately push on the boundaries of interaction designthat we learned from the physical drawing practice toprovide a more complete vocabulary in this direction thatinteraction designers can leverage.
In the future, we plan to further investigate how toincorporate other contextual information in drawing such asenvironmental lighting, as well as looking more closely
into the interactions of multiple context dimensions, suchas the relative position and distance between the pen andthe tablet beyond simple proximity. We are also interested
in applying context sensing in other creative domains suchas music.

Like physical canvases, the drawing tablet can be rotated tobe either the portrait or the landscape orientation. Thisorientation contains contextual information to infer which
type of subjects that the user may like to draw, and we canprovide digital assistance accordingly. As a simple
demonstration, our prototype offers templates of common
subjects to help the user guide their drawing. The user can
open the template library, preview them, and choose one tobe displayed in the background so that she can develop her
drawing on top of it. The available templates are filteredaccording to the current tablet orientation.

REFERENCES

1. Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G., andBuxton, W. Digital tape drawing. UIST, (1999), 161–169.
2. Bi, X., Moscovich, T., Ramos, G., Balakrishnan, R., andHinckley, K. An exploration of pen rolling for penbased interaction. UIST, (2008), 191–200.
3. Chu, N., Baxter, W., Wei, L., and Govindaraju, N.
Detail-Preserving Paint Modeling for 3D Brushes,
NPAR, (2010)
4. Fitzmaurice, G.W., Balakrishnan, R., Kurtenbach, G.,
and Buxton, B. An exploration into supporting artworkorientation in the user interface. CHI, (1999), 167–174.
5. Hinckley, K., Dixon, M., Sarin, R., Guimbretiere, F.,
and Balakrishnan, R. Codex: a dual screen tablet
computer. CHI, (2009), 1933–1942.
6. Hinckley, K., Pierce, J., Sinclair, M., and Horvitz, E.
Sensing techniques for mobile interaction. UIST, (2000),
91–100.
7. Kim, K.-E., Chang, W., Cho, S.-J., et al. Hand grip
pattern recognition for mobile user interfaces.
Innovative applications of artificial intelligence. (2006),
1789–1794.
8. Lee, J., Su, V., Ren, S., and Ishii, H. HandSCAPE: avectorizing tape measure for on-site measuring
applications. CHI, (2000), 137–144.
9. Ramos, G., Boulos, M., and Balakrishnan, R. Pressure
widgets. CHI, (2004), 487–494.
10.Song, H., Benko, H., Guimbretiere, F., Izadi, S., Cao, X.,
and Hinckley, K. Grips and Gestures on a Multi-touchPen. CHI, (2011), 1323 -1332.
11. Taylor, B.T. and Bove, J. Graspables: grasp-recognition
as a user interface. CHI, (2009), 917–926.
12. Tian, F., Xu, L., Wang, H., et al. Tilt menu: using the
3D orientation information of pen devices to extend theselection capability of pen-based user interfaces. CHI,
(2008), 1371-1380.
13. Wimmer, R. and Boring, S. HandSense: discriminating
different ways of grasping and holding a tangible userinterface. TEI, (2009), 359–362.

PRELIMINARY USER FEEDBACK

To obtain preliminary user feedback, we invited three
university art students, aged between 20-26, to try our
prototype. They had between 2-7 years of physical drawing
experience, and between 0.5-2 years of experience drawing
with digital pen and tablets. After being introduced to andfamiliarized with the prototype features, each participant
was asked to make three drawings: one still life, one
portrait, and one free creation. We observed their behavior
using the prototype, and interviewed about their experience
afterwards. All participants quickly mastered the prototype,
and were able to quickly trigger the functions they needed.
They all highly rated the naturalness, convenience, andcomfort of the system. The sketching pencil together withthe pencil sharpener was rated the highest. Interestingly,
opinions were divided around the drawing grid activated bymoving the pen away from the tablet. Some agreed that thegrid helped them in sketching the rough skeleton, which
would be especially helpful for beginners. On the other
hand, they complained that the grid was sometimes
triggered unintentionally since it appeared as soon as thepen was out of the tracking range. To address this, we
could augment our system with a finer distance sensor (e.g.
a range finder to allow higher flexibility and accuracy). We
individually asked about the naturalness of each mapping
between the contexts and interactions, and they agreed thateach of these mappings was natural to them.
CONCLUSION

This work explored how context sensing can be used toenhance the naturalness of digital pen-and-tablet drawing.
We achieved this by interpreting contextual informationthat is crucial in physical drawing practices in order tofacilitate interactions with digital drawing. The result is adigital drawing experience that more intuitively aligns withthe physical drawing practices, while at the same time4