Surfboard: Keyboard with Microphone
as a Low-cost Interactive Surface
Jun Kato1,2, Daisuke Sakamoto1, Takeo Igarashi1,2
12
JST ERATO IGARASHI Design UI ProjectGraduate School of Information Science and1-28-1, Koishikawa, Bunkyo-ku, Tokyo, 112-0002,
Technology, The University of TokyoJapan
7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan{kato | sakamoto}@designinterface.jp
takeo@acm.org
ABSTRACT

We introduce a technique to detect simple gestures of “surfing” (moving a hand horizontally) on a standard keyboard
by analyzing recorded sounds in real-time with a microphone attached close to the keyboard. This technique allows
the user to maintain a focus on the screen while surfing on
the keyboard. Since this technique uses a standard keyboard
without any modification, the user can take full advantageof the input functionality and tactile quality of his favorite
keyboard supplemented with our interface.
ACM Classification: H5.2 [Information interfaces andpresentation]: User Interfaces - Input devices and strategies.
General terms: Design, Human Factors
Keywords: Keyboard, microphone, low-cost, interactive

surface
INTRODUCTION

As a set of physical buttons laid out flat, a keyboard can bethought of as a bumpy surface. We propose a technique
named Surfboard which aims to augment the input capability of a keyboard by recording and analyzing sounds produced when the user lightly touches the keyboard andmoves his fingers horizontally over it. We named this action “surfing”. The keyboard is a standard input device andwe are not the first to augment it. Dietz et al. presented apractical pressure-sensitive keyboard [1] which provides
pressure information supplemented with pressed keys.
Block et al. presented a touch-display keyboard [2], each ofwhose key tops is augmented with a touch sensor and agraphics display on its surface. There is also a commercial
product that covers a keyboard with a multi-touch surface[3]. Unlike these previous attempts, Surfboard augments akeyboard with a monaural microphone, making its hardware
setup simple and inexpensive. It adds an additional operation modality, “surfing” to standard keyboards withoutchanging their physical properties. Simple gestures of SurfPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee.
UIST’09, October 4–7, 2009, Victoria, British Columbia, Canada.
Copyright 2009 ACM 978-1-60558-745-5/09/10...$10.00.

Figure 1. “Surfing” gesture on a keyboard

board allow the user to maintain a focus on the screen during a surfing gesture. Since surfing and typing happen insame place, the user can seamlessly continue touch typing
after surfing.
Compared with a standard keyboard operation, surfing does
not take time to find an intended key among a lot of keys. A
mouse and a touchpad are generally used for the cursor
operation and cannot be used with fingers put in the home
position of the keyboard. These comparisons suggest that
Surfboard is good for tasks which are often required during
text entry or tasks which must be done in short time when
the need arises.
SURFING ON A KEYBOARD

Surfboard can recognize whether the user is surfing on akeyboard or not (Figure 1). When the user surfs on a keyboard, it can distinguish left to right or right to left surfing
directions. We have not yet tested other motions like drawing circle or more complex shapes because they would be
difficult to execute on a keyboard which is generally horizontally long.
Surfboard monitors the user’s typing activity and its gesturerecognizer automatically stops when the user is typing
something. As a result the user need not explicitly switch
between surfing and typing. Additionally, surfing can beused in combination with pressing modifier keys like Shift,Control and Alt to simulate multi-touch gestures. For example, an application may zoom in when a Shift key istraining process, the system robustly detects the user’s surfing gestures. During the general use of Surfboard, if theclassifiers mistakenly recognize a gesture, the user can correct the last result to improve detection accuracy.
LIMITATION AND FUTURE WORK

Figure 2. Microphones attached close to keyboards

pressed and surfing is detected from left to right, and zoom
out when surfing from right to left.
PROTOTYPE IMPLEMENTATION

Our prototype implementation uses a monaural microphone
attached close to the keyboard of a personal computer. Itneeds to be attached near the right or left edge of the keyboard to distinguish surfing direction effectively. Fortunately at present, many laptop computers are equipped with amicrophone, and we confirmed that Surfboard works withseveral microphone positions (Figure 2).
The users’ surfing sounds on the keyboard are captured at asampling rate of 44.1 kHz and processed with a Fast Fourier Transform function to get amplitude information by frequencies. Static environmental noise is recorded beforehand
and is subtracted from the raw information. As shown in
Figure 3, the process of surfing recognition consists of twophases of Naïve Bayes classification. The first phase detects
whether the user is surfing or not in real-time withresampled low resolution sound data. A set of sound data is
divided into 50 frequency bands with amplitude information
varying from zero to five. The second phase starts when thefirst classifier recognizes the current sound as surfing.
Sound is recorded until the end of surfing is detected. At
the end, all recorded data is resampled along the time axis
to have normalized length which represents change insound amplitude during surfing. The recorded data isresampled into the former and the latter part, each of whichholds amplitude information of five frequency bands varying from zero to five.
The user needs to train the classifiers in two steps before
starting to use Surfboard. First, the user is asked to press
the Shift key and surf on a keyboard repeatedly for twenty
seconds. Second, the user is asked to surf from left to rightand right to left for ten times each. After this one minuteIn our future work, we will investigate use of duration information of a surfing action. It could be used to represent
the strength of command like a scaling factor in the case ofzooming. In addition, grouping multiple gestures like ashort surf after long one together as one command will provide a greater degree of freedom in the design of gesturecommands. This is similar to Scratch Input [4], which uses
scratching sounds on a flat surface, in the sense that acoustically unique gestures are recognized.
We will also explore the use of multiple microphones,
which we expect to provide more robust recognition results
through richer directional information. It might be possible
to estimate the rough position of the surfing hand similar to
PingPong Plus [5] which detects the position of a ball hitting a game table with a microphone array. Specifically, theuse of a stereo microphone should be investigated firstsince it can be easily carried out on many laptop computerswhich have capability to capture stereo sound.
This research could also benefit from improvements to its
supervised learning process. The user interface for the training process needs improving. The choice of machine learning technique directly affects the robustness of the gesturerecognition. Blowable user interfaces [6], which localizes
where the user is blowing on the screen with a monauralmicrophone, uses k-Nearest Neighbor classification. Comparing several techniques may achieve better performance.
REFERENCES

1. Dietz, P. H., Eidelson, B., Westhues, J., and Bathiche, S.
A practical pressure sensitive computer keyboard. In
Proc. of UIST’09, ACM, NY, 2009, pp.55-58.
2. Block, F., Gellersen, H., and Villar, N. Touch-display
keyboards: transforming keyboards into interactive surfaces. In Proc. of CHI’10, ACM, NY, 2010, pp.11451154.
3. FingerWorks: http://fingerworks.com/
4. Harrison, C. and Hudson, S. E. 2008. Scratch input:
creating large, inexpensive, unpowered and mobile finger input surfaces. In Proc. of UIST’08, ACM, NY,
2008, pp.205-208.
5. Ishii, H., Wisneski, C., Orbanes, J., Chun, B., and Paradiso, J. 1999. PingPongPlus: design of an athletictangible interface for computer-supported cooperativeplay. In Proc. of CHI’99, ACM, NY, 1999, pp.394-401.
6. Patel, S. N. and Abowd, G. D. 2007. Blui: low-cost localized blowable user interfaces. In Proc. of UIST’07,ACM, NY, 2007, pp.217-220.

Figure 3. Process of gesture recognition