Interaction Technique for a Pen-Based Interface Using
Finger Motions
Yu Suzuki, Kazuo Misue, and Jiro Tanaka
1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8573, Japan{suzuki,misue,jiro}@iplab.cs.tsukuba.ac.jp

Abstract. Our research goal is to improve stylus operability by utilizing thehuman knowledge and skills applied when a user uses a pen. Such knowledge
and skills include, for example, the way a person holds a pen to apply agenerous amount of ink to draw a thick line with a brush pen. We propose aform of interaction, Finger Action, which uses input operations applying suchknowledge and skills. Finger Action consists of five input operations: gripping,thumb tapping, index-finger tapping, thumb rubbing, and index-finger rubbing.
In this paper, we describe Finger Action, a prototype pressure-sensitive stylus
used to realize Finger Action, an application of Finger Action, and anevaluation of the practicality of Finger Action.
Keywords: Pen-based Interface, Finger Motion, Pen Grip, Pressure Sensor.

1 Introduction
Much research has been done on the application of pre-existing human knowledge
and skills for human-computer interaction (HCI) technology [1]. An advantage ofthese approaches is that they enable people to operate computers after a modest
amount of learning or training. We believe that such an approach is important to further the development of HCI technology.
Many people have used a pen from a young age and the pen is one of their favorite
tools. We think that a pen-shaped interface stylus is one of the best forms of interface
because a stylus is similar to a real pen and its use takes advantage of existing humanknowledge and skills. However, the current stylus does not offer high operability
because it does not fully utilize the freedom allowed by a pen. The input information
used with the current stylus consists of only coordinates indicating where the tip is on
a display contact, the pressure applied to the tip, and ON/OFF as selected with a barrel button.
The purpose of our research has been to improve the operability of the pen-based
interface. Previously, we studied use of the positions and postures of the stylus in theair [2]. In this research, we observed the behaviors of people when they used pens,and found that most made intricate finger motions when using a pen. These finger
motions can be considered acquired knowledge and skills related to pen use, and we
think there are other knowledge and skills concerning pen use in addition to those wehave observed. We call this kind of knowledge and skills “pen-experience”.
J.A. Jacko (Ed.): Human-Computer Interaction, Part II, HCII 2009, LNCS 5611, pp. 503–512, 2009.
© Springer-Verlag Berlin Heidelberg 2009

504

Y. Suzuki, K. Misue, and J. Tanaka

In our current work, we have tried to use pen-experience to improve pen-based interfaces. In this paper, we examine whether we can use finger motions to operate a
pen-based interface and how such finger motions can be applied.

2 Related Work
The purpose of our research is to improve stylus operability. Our approach is to use
pre-existing human knowledge and skills. In this section, we clarify the position of
our research by introducing related work from two aspects: that of research having thesame goal as our research and that of research aimed at applying pre-existing humanknowledge and skills.
2.1 Research Aimed at Improving Stylus Operability
Other research aimed at improving stylus operability has been done by various researchers.
Compared to mouse-click operations, stylus tapping operations are difficult to docorrectly. Therefore, research has been done to develop circular rather than linear
menus. Hopkins developed PieMenu [3] as a menu interface which enables easy operation with a stylus. Other menu interfaces which use a circular menu like PieMenu
include Marking Menu [4] and FlowMenu [5].
An interaction technique called “crossing” was proposed as a way to invoke commands by using a stylus when operating GUIs [6]. While it is difficult to correctly tap
a small target with a stylus, it is easy to cross a target when the stylus is in contact
with the display. Crossing takes advantage of this ease.
Smith et al. focused on the difficulty of operating a scroll bar with a stylus, andtheir research was aimed at improving stylus operability in this regard [7, 8]. Theydeveloped a tool which recognizes a clockwise gesture on the display. When using
this tool, a user can scroll anywhere on the display.
The approach of these studies was to find ways to overcome the difficulty of tapping a small area with a stylus. In contrast, our approach is based on pre-existing
human knowledge and skills.
2.2 Research Using Pre-existing Human Knowledge and Skills
Tangible Bits [9], proposed by Ishii et al., can be understood as the application of preexisting human knowledge and skills. One concept underlying Tangible Bits is thecoupling of bits and atoms in a way that allows a person to apply pre-existing knowledge and skills.
Mohler et al. developed an interaction technique for moving through virtual spaceby walking on a treadmill [10]. While a user can use a mouse or joystick as a means
to move through virtual space, the user has to train operations before using them. By
applying walking actions to moving through virtual space, a person can move throughvirtual space without any training because it is based on moving through real space bywalking.
Rekimoto proposed using the tilt of devices as an interaction method for small display devices [11]. For example, a user can change the viewpoint in a map browser byInteraction Technique for a Pen-Based Interface Using Finger Motions

505

tilting the device. Such control is intuitive to the user because it follows physical laws
understood by the user.
Depth perception in virtual 3D space is represented on 2D display by using atmospheric colors, fog, shadows and so on [12]. These techniques rely on the user’s
knowledge that distant objects look blurred.
These studies have attempted to apply pre-existing human knowledge and skills tointerfaces of VR, AR and small devices. In contrast, we aim to apply them to a penbased interface.

3 Proposed Interaction Technique
3.1 Using Pen-Experience
A benefit of applying pre-existing human knowledge and skills to HCI is that a user
can use a computer with minimal learning or training. The pen-based interface imitates the use of a real pen and paper and uses pen-experience for operations. Therefore, the user can use a pen-based interface without having to learn how to operate theinterface.
By focusing on the pen and typical human actions when using one, we found that
there are other pen-experiences. For example, when drawing thick lines with a fudepen1 , the user applies more pressure to make more ink come out of the pen. This
small change in the amount of power applied to the fingers is rarely used in pen-based
interfaces. We aim to improve the stylus operability by using such actions based on
pen-experience.
3.2 Finger Action
There are various finger motions related to pen-experience. To be used as input operations of a pen-based interface, these motions must meet two conditions: be simple forthe user to perform and not physically strain the user. The following motions satisfy
these two conditions:
A)
B)
C)

A motion to more strongly grasp the stylus grip
A motion to lightly tap the grip with the thumb
A motion to lightly tap the grip with the index finger.

Other motions are also physically possible, such as rubbing motions. Althoughthese are not always incidental motions when using a pen, we also experimentallytested two other motions:
1. A motion to rub the grip with the thumb
2. A motion to rub the grip with the index finger.
We call these five motions finger actions (Fig. 1). Finger actions can be broadlyclassified into three motions: (A) gripping, (B) tapping, and (C) rubbing. Tapping and

1

A fudepen is a kind of brush pen. Because a fudepen is made from soft material, the user canchange the amount of ink applied and the line thickness by applying more or less pressure.

506

Y. Suzuki, K. Misue, and J. Tanaka

rubbing can each be separated into two types: (1) thumb motion and (2) indexfinger motion. We refer to these as (B-1) thumb tapping, (B-2) index-finger tapping,(C-1) thumb rubbing, and (C-2) index-finger rubbing. The interaction technique wehave established based on these actions for a pen-based interface is called Finger
Action.

(A) Gr(BB) TT(C) IT(D) TR(E) IR

Fig. 1. Five motions of Finger Action: (A) – (E) respectively show gripping, thumb tapping,index-finger tapping, thumb rubbing, and index-finger rubbing

4 Development of a Pressure-Sensitive Stylus
We have developed a pressure-sensitive stylus that enables a user to use Finger Action. In this section, we describe this stylus.
4.1 Requisite Data to Realize Finger Action
A human uses three fingers to control a pen (the thumb, index finger and middle finger), so the pressure-sensitive stylus must be able to detect these three fingers. In
other words, we need to be able to sense the three points where these fingers makecontact with the stylus grip. In addition, to apply the rubbing operation with either the
thumb or the index finger, we have to ensure the stylus can sense the vertical movement of these two fingers.
4.2 Pressure-Sensitive Stylus
Figure 2 shows the pressure-sensitive stylus. We use FSR4022 pressure sensors todetect pressure applied to the stylus grip. The stylus is equipped with five of thesesensors, each with a diameter of about 18 mm. A sensor’s electrical resistancedecreases as the applied pressure increases. We used two of the sensors for the
thumb, two for the index finger, and one for the middle finger. We use a PICBASIC microcomputer to read the sensors’ output, and the gripping power isanalyzed in real time. The time resolution when reading out sensor values is
200 ms.
A user does not need any special instruction to use the pressure-sensitive stylus because the sensors are placed where a person naturally holds a pen.

2

Interlink Electronics, Inc.

Interaction Technique for a Pen-Based Interface Using Finger Motions

507

Fig. 2. Pressure-Sensitive Stylus

5 Evaluation of Effectiveness
Gripping, thumb tapping, index-finger tapping, thumb rubbing, and index-finger rubbing are common actions when a person actually uses a pen, are simple to do, and are
physically easy to perform. To determine whether these actions are suitable for input
operations, though, we did an experiment to investigate the usefulness of Finger Action as a pen-based interface.
5.1 Outline of the Experiment
In the experiment, we measured the ease of the actions for users and the physicalstrain imposed upon the user when performing the actions.
The experiment consisted of two tasks. In task 1, participants did operations while
concentrating on performing the five input operations. A few seconds after the experiment began, a short trigger sound was presented. The participants were to do an
input operation when hearing the sound. The sound was presented ten times in eachtrial with the time interval randomly varied between 3-6 seconds. The interval variation was to prevent the participants from guessing the timing of the trigger. Eachparticipant performed this trial five times for each input operation, so in task 1 a participant performed every input operation fifty times.
In task 2, participants did the same operations as in task 1 while drawing characters, a picture, or something similar. That is, participants did the trial independently intask 1, but they did the trial while doing other work in task 2. After finishing tasks 1
and 2, the participants completed a questionnaire where they evaluated the operabilityof the stylus and the physical strain of using it on a scale of one to five. (A score of 5
indicated the best operability or the least strain and a score of 1 indicated the worst
operability or the heaviest strain.)
There were eight participants: seven males and one female. Because none of theparticipants were familiar with Finger Action, we spent five minutes explaining how
to use Finger Action. The participants then practiced all of the actions a few times.
Each task in the experiment took about twenty minutes, and there was a break between the two tasks.

508

Y. Suzuki, K. Misue, and J. Tanaka

We evaluated the ease of the actions according to the average measured time foreach operation and the questionnaire responses, and evaluated the physical strain
according to the questionnaire responses.
5.2 Results
We show the mean time needed to perform each input operation and the standard
deviation in Fig. 3, and the questionnaire results in Fig. 4.
First, we consider the ease of each action. In some cases, sensing errors or participants’ operation errors did not allow the participants to perform the intended actions.
Such errors are not included in these results.
Human cognitive processing and the processing time of the prototype system affected the results. Human cognitive processing begins with recognition of a stimulus
through a perception system, which is followed by analysis of the detected signal’s
meaning through the cognitive system, and then transmission of an action to be taken to
the motor system. In our experiment, this processing sequence took 300-400 s [13]. As
mentioned, the time resolution in our prototype system when reading out sensor values
is 200 ms. When the system detects tapping or rubbing, the average delay is 200 ms
because the system uses two contiguous values. Therefore, we considered a good result
to be 500-600 ms for gripping and 700-800 ms for tapping or rubbing.
In the task 1 results, the mean value for gripping slightly exceeded 600 ms and fortapping it slightly exceeded 800 ms, so the amount of delay was acceptable. In thequestionnaire results, the scores for the operability of gripping and tapping exceeded
the midpoint score of 3, indicating that gripping and tapping were easy actions toperform independently.
Comparing the task 1 results to the task 2 results, we see that the average delay
times were 100-200 ms longer for task 2 for all operations. In task 2, the participants
had to switch from their current work to another type of work to perform the required
finger action, and this probably is the reason for the greater delay. The questionnaire
results again showed that the average response for the operability of the gripping andtapping operations exceeded the midpoint score of 3. Therefore, although the time forFig. 3. Mean time (ms) for performing each
input operation and the standard deviation

Fig. 4. Questionnaire results

Interaction Technique for a Pen-Based Interface Using Finger Motions

509

operation was about 100 ms to 200 ms longer than when performing the gripping andtapping operations independently, the operations were easy to perform.
Next, we consider the physical strain. The average questionnaire responses forgripping and tapping exceeded 3.5 in both task 1 and task 2. Therefore, gripping andtapping operations can be done without undue physical strain when performing theoperations either independently or dependently.
Last, we consider the number of input errors (Fig. 5). There was a big difference
between the average number of input errors in tasks 1 and 2 for all operations. A t-test
at a 5% significance level for the average number of input errors showed a significant
difference for tapping (thumb tapping: p=0.0002, index-finger tapping: p=0.0021).
Therefore, the number of input errors is likely to increase when performing the tapping operation while doing other work.

Fig. 5. Average number of input errors

To summarize the results, gripping and tapping appear to be useful operations
whether work is being done independently or dependently. However, many operation errors are likely to occur when a user performs the tapping operation while
also doing other work. This might be because the user has to lift his/her finger off
the stylus to perform the tapping. We propose a way of mapping between Finger
Action and the operations of applications: gripping should be mapped as an operation which can be done while the user is engaged in another operation at thesame time, while tapping should be mapped as an operation which can be done
independently.
Improving the rubbing operability will be part of our future work.

6 Application of the Pressure-Sensitive Stylus
We developed a paint tool, BrushPaint, as an application of the pressure-sensitive
stylus. When using BrushPaint, a user can interact with a computer in a way that is
like using a real pen. In this application, we assigned each of the gripping, tapping
and rubbing operations to a specific function:

510

Y. Suzuki, K. Misue, and J. Tanaka

Fig. 6. BrushPaint screenshot• Gripping: changing the amount of ink• Index-finger tapping: applying ink• Index-finger rubbing: changing the ink color
We mapped the operations and functions in a way that makes it easy for the user toimagine the result of each operation. In addition, we considered our findings regarding the users’ aptitudes for each operation, as discussed in Sect. 5.
We applied the fudepen metaphor to gripping in BrushPaint. When using a
fudepen, the ink flow increases when a stronger grip is used to apply more pressure, so the stroke width changes depending on the strength of the grip. We implemented a stroke width change function in BrushPaint that likewise allows theuser to change the width by adjusting the strength with which the stylus is gripped.
We also considered that gripping more strongly or gently would be easy for a user
to do.
We similarly applied the brush pen metaphor to index-finger tapping. When auser taps a brush which has a lot of ink, the ink drops onto the paper, and theamount of dropped ink changes depending on the tapping strength. When the user
taps the pressure-sensitive stylus, some ink drops onto the virtual canvas, andagain the amount of dropped ink depends on the tapping strength. The distancefrom the stylus to the dropped ink changes randomly. We also considered thatfewer tapping operation errors are likely to occur when the operation is performed
independently.
We assigned an ink changing function, which might be the most frequently usedfunction, to the index-finger rubbing operation. The hue of the ink changes at a constant rate every time index-finger rubbing is done.
In general paint tools, a user has to use menu operations even if the intended operation is very simple. However, the use of Finger Action in BrushPaint allows the user
to use three functions without resorting to menu operations. In general, since menu
operation using a stylus is awkward, Finger Action should make operations with astylus much easier.

Interaction Technique for a Pen-Based Interface Using Finger Motions

511

7 Conclusion and Future WorkIn this paper, we have proposed a novel interaction technique, called Finger Action,that is based on pen-experience. Pen-experience means a user’s pre-existingknowledge and skills concerning the use of a pen. Finger Action consists of five input
operations: gripping, thumb tapping, index-finger tapping, thumb rubbing, and indexfinger rubbing. We developed a pressure-sensitive stylus equipped with pressure
sensors to realize Finger Action. Using the pressure-sensitive stylus, we did an experiment to evaluate the usefulness of Finger Action as input operations for the stylus.
We found that while gripping and tapping are useful, rubbing does not work well in
the current implementation. In addition, we introduced BrushPaint, an application of
the pressure-sensitive stylus.
As our future work, we plan to refine the pressure-sensitive stylus, and then conduct an experiment comparing Finger Action with other stylus interaction techniques.
Acknowledgment. This study was supported in part by the Global COE Program on“Cybernics: fusion of human, machine, and information systems.”

References
1. Jacob, R.J., Girouard, A., Hirshfield, L.M., Horn, M.S., Shaer, O., Solovey, E.T., Zigelbaum, J.: Reality-Based Interaction: A Framework for Post-WIMP Interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2008), pp.
201–210 (2008)
2. Suzuki, Y., Misue, K., Tanaka, J.: Pen-based Interface Using Hand Motions in the Air. The
IEICE Transactions on Information and Systems E91-D(11) (2008)
3. Hopkins, D.: The Design and Implementation of Pie Menus. Dr. Dobb’s Journal 16(12),16–26 (1991)
4. Kurtenbach, G., Buxton, W.: The Limits Of Expert Performance Using Hierarchic Marking Menus. In: Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems (CHI 1993), pp. 482–487 (1993)
5. Guimbreti´ere, F., Winograd, T.: FlowMenu: Combining Command, Text, and Data Entry.
In: Proceedings of the 13th Annual ACM Symposium on User Interface Software andTechnology (UIST 2000), pp. 213–216 (2000)
6. Accot, J., Zhai, S.: More than dotting the i’s - Foundations for crossing-based interfaces.
In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI 2002), pp. 73–80 (2002)
7. Smith, G.M., Schraefel, M.C.: The Radial Scroll Tool: Scrolling Support for Stylus- or
Touch-Based Document Navigation. In: Proceedings of the 17th Annual ACM Symposium
on User Interface Software and Technology (UIST 2004), pp. 53–56 (2004)
8. Schraefel, M. C., Smith, G., Baudisch, P.: Curve Dial: Eyes-Free Parameter Entry forGUIs. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2005), pp. 1146–1147 (2005)
9. Ishii, H., Ullmer, B.: Tangible Bits: Towards Seamless Interfaces between People, Bits andAtoms. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 1997), pp. 234–241 (1997)

512

Y. Suzuki, K. Misue, and J. Tanaka

10. Mohler, B.J., Thompson, W.B., Creem-Regehr, S.H., Willemsen, P., Herbert, L., Pick, J.,
Rieser, J.J.: Calibration of Locomotion Resulting from Visual Motion in a TreadmillBased Virtual Environment. ACM Transactions on Applied Perception (TAP) 4(1) (2007)
11. Rekimoto, J.: Tilting Operations for Small Screen Interfaces. In: Proceedings of the 9th
Annual ACM Symposium on User Interface Software and Technology (UIST 1996), pp.
167–168 (1996)
12. Bowman, D.A., Kruijff, E., LaViola, J.J., Poupyrev, I.: 3D User Interfaces: Theory andPractice. Addison Wesley Longman Publishing Co., Inc. (2004)
13. Okada, K., Nishida, S., Kuzuoka, H., Nakatani, M., Shiozawa, H.: Human Computer Interaction. Ohmsha, Ltd. (2002) (in Japanese)