Haptic Feedback for Pen Computing:Directions and Strategies
1

Ivan Poupyrev 1
Interaction Lab, Sony CSL
3-14-13 Higashigotanda,
Shinagawa, Tokyo141-0022 Japan
poup@csl.sony.co.jp

2

Makoto Okabe 2, 1
Department of Computer Science,
University of Tokyo,
7-3-1 Hongo, Bunkyo,
Tokyo 113-8654, Japan
makoto21@ui.is.s.u-tokyo.ac.jp

ABSTRACTWe seek to improve the experience of using pen computing
devices by augmenting them with haptic, tactile feedback
displays. We present the design of the haptic display forpen computers, and explore interaction techniques that allow users to feel GUI elements, textures, photographs andother interface elements with a pen. We discuss research
directions in haptic displays for pen devices and report results of an early experimental study that evaluated the benefits of tactile feedback in pen computing.
Author Keywords

pen input, haptic feedback, experimental studies, Fitt’s law
ACM Classification Keywords

H5.2. [Information interfaces and presentation (e.g., HCI)]:
User interfaces – Graphical user interfaces (GUI), Input
devices and strategies, Interaction styles, Screen design.
INTRODUCTION

Pen-based and tablet computers are becoming increasingly
popular in home, businesses and industrial use. They are
attractive because of their intuitive interaction style. Insteadof using indirect devices such as a mouse, the user can use apen to push graphical buttons, drag sliders, write and sketch
directly on a screen, just like on a piece of paper. Furthermore, recently implemented in the form of tablets, pen
computers are compact and easy to use.
Although the history of pen input stretches back as far as
the RAND tablet, developed in the 50s [10], until recently
pen-based devices were used mostly by users of computer
graphics systems: designers, artists and architects. Recently,
however, the variety and availability of devices that support
pen input have grown with introduction of PDAs and tablet
PCs. With further improvement of technology, the popularity of pen-based computing will certainly continue to growin the future.

Shigeaki Maruyama 3, 13
Micro Device Center,Sony EMCS,
2-15-3 Konan, Minato,
Tokyo 108-6201 Japanshigeaki.maruyama@jp.sony.com

problem of lack of haptic feedback when interacting withtouch and pen devices was noted by Buxton as early as
1985 [5]. The sense of touch plays a crucial role in our interaction with the physical world, as it allows us to perceive
object textures, recognize shapes and objects by touch, andallows for effective blind manipulation. As a familiar example, we can drink coffee while reading a newspaper, operating the coffee cup largely by touch.
We aim to investigate how these powerful haptic processing
capabilities can be used to enhance interaction with pencomputers. We present the design of the tactile display forpen computers, interaction techniques and results of anearly experimental study that evaluated the benefits of tactile feedback in pen computers. We also discuss the directions and strategies for future exploration of tactile feedback in pen devices.
BACKGROUND AND PREVIOUS WORKPen interaction is similar to touch screen interaction. However, pen input covers a wider range of devices, since it canbe used both with touch screens and specialized pen input
devices, such as Wacom™ interactive pen displays. The
pen is very effective when fast, fluid and high precision
two-dimensional input is required. Therefore artists anddesigners often use pen-enabled displays in painting and 2D
layout applications. Pens are also used in mobile devices,such as tablet computers, as a replacement for keyboard, orwith PDAs, where pen allows effective interaction despite
their small screens.

We seek to improve the experience of using pen devices byaugmenting them with haptic and tactile feedback. The

Copyright is held by the author/owner(s).
CHI 2004, April 24–29, 2004, Vienna, Austria.
ACM 1-58113-703-6/04/0004.

Figure 1: Prototype of tactile pen computer

a.

b.

e.

f.

c.

d.

g.

h.

Figure 2: Pen interaction were extended with tactile feedback

There have been few attempts to combine a pen with haptic
feedback devices. For example, pen-style force-feedback
displays have been used for virtual reality interaction [4]and to simulate a feel of real brushes in paint programs [3].
Unlike this work, we focus on designing tactile interfaces
for everyday tablet and pen computers, rather than 3D userinterfaces or stand-alone haptic interactive systems.

the display. We then enhanced 2D pen interaction techniques, such as GUI and drawing, with tactile feedback andinformally evaluated these enhancements by demonstratingthem to colleagues and visitors. Finally, after refinement,
we conducted experimental studies that investigated theeffect of tactile feedback in pen tapping and drawing tasks.

Active Click [7] was the earliest reported tactile interface
for touch screens that could also be used with a pen. Its offthe-shelf coil-type actuators, however, were too large forthin tablet devices and its tactile vocabulary was limited toa narrow range of vibration frequencies. Our tactile display
provides for independent control of vibration amplitude andfrequency. This allows for an infinite variety of tactile feelings for various GUI elements and interaction scenarios.

We used TouchEngine haptic feedback technology to add
haptic response to the screen. TouchEngine is comprised ofthin custom-built piezoelectric actuators, control hardware,and software. It is described in detail in [11, 12]. The basic
display design used here is identical to [11], except thatlarger (30×5mm) actuators were used. Here we provide
only the most basic information on the design.

We are not aware of previous work that systematically investigated tactile interfaces for pen computers. For desktop
interaction, haptic mice have been developed either by adding a solenoid [2], voice coils [9] or motors [1]. In contrast,
our work investigates haptic feedback for pen-based devices that are operated without a mouse.

Haptic display for pen

Four TouchEngine actuators are embedded in the corners ofdisplay in between the LCD and the thin protective glass
panel on top. We attached the glass panel to the base withthin silicon adhesives that let it move slightly, but kept
glass safely attached to the base. When the piezoelectric
actuators bend in response to the electrical signal, they push
the glass outwards and the user can feel vibration throughthe pen which is touching the screen. The amount of bending is proportional to the signal amplitude that is generated
by the TouchEngine control board. Currently, we are using
the simple pulses of different intensities.

This project is part of a long-term research effort that investigates haptic displays for everyday devices using theTouchEngine™ tactile technology that we have developed[13]. Previously, we focused on small handheld computers.
For example, [10] reports the design of “full body” tactile
displays for handheld devices that provide expressive ambient tactile notifications to mobile users. In [11] we enhanced PDA touch screens with tactile feedback, so thatusers can feel basic GUI elements, such as buttons and sliders. This paper further develops this direction of research.

Pen-based haptic interaction

HAPTIC FOR PEN COMPUTERS

GUI interaction

TouchEngine actuators were embedded into a stationary 15inch LCD monitor of Sony VAIO LX personal computer(Figure 1) that has Wacom pen-input technology built into

We added tactile feedback to GUI elements, such as graphical buttons, so that they would “click” when user pushedthem, sliders, that provided a short tactile impulse each

We explored three strategies for tactile interaction with pen
computers: tactile GUI, tactile information perception, andtactile feedback for active input. This section discusses several simple examples of tactile interfaces that we developedand informally evaluated.

time the user scrolled a line, and text, where selecting acharacter was enhanced with a tactile click (Figure 2e, f). In
informal evaluations, the users were very positive abouttactile feedback. We observed that users strongly preferredtactile feedback when it was combined with a gesture, e.g.
dragging sliders or selecting text. This preference may be
explained within Gibson’s active touch paradigm [8].
Feeling data

Tactile display may allow the user to feel visual data via
the cutaneous sensory channel (e.g. feeling bumps whendrawing across simple textures with a pen, or feeling cracks
in the photograph of a desert). This is similar to how we canfeel physical textures in the real world (Figure 2d, g). We
also added tactile feedback to character animation. Whenthe user touches a character with the pen, the animation istriggered and the user can feel the character motion; similar
to how we can feel a pulse beating in a hand (Figure 2h).
The users reacted positively to these interfaces by sayingthat the feelings were very natural and enjoyable. We
found, however, that increasing the complexity of textures
seemed to reduce the effect of a haptic feedback and ineven some case made interaction more confusing. We hypothesize that with complex visual textures the users could
not easily correlate the image and tactile feedback, particularly when they draw quickly and the latency of pen input
was evident. Designing tactile interfaces for data perception, therefore, is not straightforward. Spatial and temporalcorrelation between visual and haptic feedback in dynamic
gestures must be explored in more depth. Techniques tocope with lag are also very important. One technique is topre-compute several tactile maps for different speeds of pen
movement and use them instead of the original image fortriggering tactile events.
Tactile feedback for active pen input

There is an inherent sense of satisfaction when drawing
with a pen or pencil. The feel of the paper, as the pen tip
moves over it, creates the inherently physical and intimate
feeling of drawing. Our discussions with artists and designers suggest that this physicality assists the artist’s creative
process.
We added tactile feedback for drawing and sketching operations by having a single tactile pulse produced each time
the pen crossed a pixel (Figure 2b). The strength of thepulse was correlated with the pen pressure so that stronger
pressure resulted in stronger tactile feedback. We also explored tactile feedback in vector-style graphics manipulation (e.g. feeling control points on the Bezier curve (Figure2a)). Indeed, in complex drawings the density of control
points is often very high and tactile feedback might be useful for improving selection precision. Finally, we combined
tactile feedback with constraints such as grids and alignments. In one example, the user would feel the underlyinggrid while manipulating a line’s endpoint in snap-to-gridmode (Figure 2c).
Active input enhanced with tactile feedback was most appreciated by the users. In particular, haptic constraints weremet with delight, as they felt similar to a pen hitting aphysical groove or guide. Combining constraints with tactile feedback in pen computing is an interesting direction
for further exploration. Even though users enjoyed tactile
feedback during the drawing task, the tactile sensation was
not similar to the feel of real pen and paper. We also observed that since vibration frequency depends on drawing
speed, rapid gestures were uncomfortable when the vibration became too strong. The user appreciated softer, gentler
tactile feedback more. Factoring gesture speed into the design of tactile feedback is essential in creating efficient andenjoyable user interfaces. At this point, we have a limited
understanding of the relationship between gesture speed,tactile feedback and resulting user experience. Finally, weobserved that all users universally liked having feedback
strength change in proportion to the pen pressure.
EXPERIMENTAL EVALUATION

Tactile feedback opens a new interface design space andbefore embarking on its exploration, we wanted to understand benefits of the tactile feedback in pen interaction andits most promising areas for investigation.
Tasks and experimental design

We designed experiments according to the ISO 9241 Part 9[6] that is based on Fitt’s experimental paradigm. Two tasks
were evaluated. First, in the tapping task, the subjects repeatedly tapped on strips of width W separated by distanceA. The target was indicated by blue color. In the tactile
condition, a tactile pulse was provided when the target was
successfully selected versus no tactile feedback in the normal condition.
Second, in the drawing/dragging task, the subjects drew aline with a pen from a starting point to the target stripe. To
accomplish the task, the subjects had to release the pen
within the target stripe. We studied three conditions: normal (i.e. no tactile feedback), a single-impulse feedback
when the user crossed the target strip edge, and continuous
feedback (i.e. after the pen entered the target strip, a pulse
would be provided for each pixel the pen crosses, resulting
in continuous tactile feedback to pen movement).
Following [6], we used 2, 5 and 10 mm target widths and40, 80 and 160 mm distances between targets. Twelve malesubjects 24 to 34 years old were recruited from the laboratory pool. A within-subject, repeated-measures experimental desugn was used where each subject completed 10 trials
for each width/distance combination, resulting in 90 trials
for each experimental condition. The order of conditions
and trials were randomized. The experiments started withtraining and finished with a questionnaire to evaluate subjective preferences. The survey asked the participants torank conditions using the Lickert scale ratings from 1(worst) to 5 (best). Each experiment took about 40 minutes.
The next section presents the main results of the experiments.
Experimental procedure and results

Tactile feedback did not improve the user performance inthe tapping task since mean completion times were almost
identical for 5 and 10 mm targets. There was some variability for 2 mm targets, but it was not statistically significant

Figure 3: Experimental results: a) mean times in tapping task; b) mean tines in drawing task collapsed across amplitudec) scatter plot of time versus Fitt`s index of difficulty with linear regression lines for different tactile conditions

with the number of subjects we tested (Figure 3a). Bandwidth, estimated using Fitt’s law, was identical in tactileand normal conditions — 7.14 bit/sec. We observed thatsubjects often hit the target rather then touch it. hence thecontact between the pen and screen was very short. In fact,
during the training, some subjects initially did not evennotice the tactile feedback. Feedback had a greater impact
for small targets, where size forced subjects to tap slowly.
The effect of feedback in selecting very small targets could
be an interesting future research direction.
Tactile feedback improved user performance in the drawing
task by a statistically significant amount. There was significant interaction between feedback, amplitude and target
width and the effect of feedback was strongest for smaller
targets (Figure 3c). The subject was about 18% faster for 2mm targets when averaged across all distances. Assumingthat the dragging falls under Fitt’s law [6], the bandwidth
could be estimated as 5.9 and 4.8 bit/sec for tactile and notactile conditions respectively. Figure 3b demonstrates thatas the difficulty of task increases the benefit of tactile feedback over no feedback conditions increases. We were surprised to find that the two modes of feedback that we studied resulted in almost identical performance. This indicatesthat just providing tactile feedback on the target boundary
might be sufficient, although subjects preferred continuous
feedback (4.2) to the feedback on the edge (3.5).
The experimental results suggest that the combination ofactive gesture with tactile feedback yields significantly better results then for the simple tapping task. This is also supported by the survey of the users’ subjective preferences.
On average, subjects rated tactile feedback in the drawing
task as 3.9 versus 2.5 for no feedback on a 1 to 5 scale. In
the tapping task the ratings where 3.4 and 2.4 respectively.
We believe that these tasks and conditions approximate abulk of the interactions in pen computing (i.e. selecting
targets of various sizes separated by differing distances,dragging icons and scroll bar handles). Therefore, the results of the experiments can be generalized to traditional
GUI applications.
CONCLUSIONSWe presented an overview of haptic interfaces for pen
computing, a new area for interaction research. The mostbasic interaction scenarios have been discussed, prototyped,and evaluated. The experiments that we conducted demonstrated the value of tactile feedback and indicated directions
for future improvements and investigations. In the future
we will conduct more extensive experimental evaluation inorder to confirm the reported experimental results.
REFERENCES
1. Immersion Corp. www.immersion.com.
2. Akamatsu, M. and S. Sato, A multi-modal mouse with tactile
and force feedback. International Journal of Human-Computer
Studies, 1994. 40(3): p. 443-453.
3. Baxter, B., V. Scheib, M. Lin, and D. Manocha. DAB: interactive haptic painting with 3D virtual brushes. Proceedings ofSIGGRAPH. 2001: ACM: pp. 461-468.
4. Buttolo, P. and B. Hannaford. Pen-based force display forprecision manipulation in virtual environments. Proceedingsof VRAIS. 1995: IEEE: pp. 217-224.
5. Buxton, B., R. Hill, and P. Rowley. Issues and Techniques inTouch-Sensitive Tablet Input. Proceedings of SIGGRAPH'85.
1985: ACM: pp. 215-224.
6. Douglas, S., A. Kirkpatrick, and S. MacKenzie. Testing pointing device performance and user assessment with ISO 9241,
Part 9 standard. Proc of CHI. 1999: ACM: pp. 215-222.
7. Fukumoto, M. and S. Toshiaki. ActiveClick: Tactile Feedbackfor Touch Panels. Proceedings of CHI'2001, Extended Abstracts. 2001: ACM: pp. 121-122.
8. Gibson, J., Observations on active touch. Psychological review, 1962. 69: p. 477-491.
9. Hughes, R. and A. Forrest. Perceptualisation using a tactilemouse. Proc. of Visualization'96. 1996: IEEE: pp. 181-188.
10. Meyer, A., Pen computing: A technology overview and a vision. ACM SIGCHI Bulletin, 1995. 27(3): p. 46-90.
11. Poupyrev, I. and S. Maruyama. Tactile interfaces for small
touch screens. Proc of UIST. 2003: ACM: pp. 217-220.
12. Poupyrev, I., S. Maruyama, and J. Rekimoto. Ambient Touch:
Designing tactile interfaces for handheld devices. Proceedings of UIST'2002. 2002: ACM: pp. 51-60.
13. Poupyrev, I., S. Maruyama, and J. Rekimoto. TouchEngine: Atactile display for handheld devices. Proceedings of CHI
2002. 2002: ACM: pp. 644-645.