Enhancing Input On and Above theInteractive Surface with Muscle Sensing
Hrvoje Benko1, T. Scott Saponas1,2, Dan Morris1, and Desney Tan1
1
2
Microsoft Research
Computer Science and Engineering Dept.
Redmond, WA, USA
University of Washington,{benko, dan, desney}@microsoft.com
Seattle, WA, USAssaponas@cs.washington.edu
ABSTRACT

Current interactive surfaces provide little or no information about which fingers are touching the surface, theamount of pressure exerted, or gestures that occur when notin contact with the surface. These limitations constrain theinteraction vocabulary available to interactive surface
systems. In our work, we extend the surface interaction
space by using muscle sensing to provide complementary
information about finger movement and posture. In thispaper, we describe a novel system that combines muscle
sensing with a multi-touch tabletop, and introduce a seriesof new interaction techniques enabled by this combination.
We present observations from an initial system evaluationand discuss the limitations and challenges of utilizing
muscle sensing for tabletop applications.
Author Keywords

Surface computing, tabletops, muscle sensing, EMG.
ACM Classification

H.5.2 [Information interfaces and presentation]: User
Interfaces. – Input devices and strategies; Graphical userinterfaces.
INTRODUCTION

Interactive surfaces extend traditional desktop computing
by allowing direct manipulation of objects, drawing on our
experiences with the physical world. However, the limited
scope of information provided by current tabletop
interfaces falls significantly short of the rich gestural
capabilities of the human hand. Most systems are unable todifferentiate properties such as which finger or person istouching the surface, the amount of pressure exerted, or
gestures that occur when not in contact with the surface.
These limitations constrain the design space and interaction
bandwidth of tabletop systems.
In this paper, we explore the feasibility of expanding theinteraction possibilities on interactive surfaces by sensing
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
to republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee.
ITS '09, November 23-25 2009, Banff, Alberta, Canada
Copyright © 2009 978-1-60558-733-2/09/11... $10.00

BioSemi EMG
Monitor/Amplifier

Forearm
sensors

Microsoft Surface
Figure 1. Our system uses electromyography(muscle activity) sensors placed on the forearm toinfer finger identity, estimate finger pressure, andallow off-surface gestures.
muscle activity via forearm electromyography (EMG).
EMG allows us to infer additional information about eachcontact with an interactive surface, and provides novel
information about hand and finger movement away fromthe surface. We employ muscle sensing in combination
with the contact sensing of a standard multi-touch tabletop(Microsoft Surface) and introduce novel interactions thatemerge from this combination of sensor streams.
As demonstrated in previous sensor fusion work, thecombination of multiple complementary streams can often
be greater than sum of the parts [8,10,15]. For example, in
our work, we use muscle sensing to determine which finger
is in contact with a surface, assess the level of pressure
exerted by the user while they are pressing down, and evendetect activity when a user’s hand is not in contact with thesurface. Combining these sensing modalities allows us to
explore finger-specific input, pressure-sensitive interaction,and free-space gestures that complement traditional onsurface interactions.
The contributions of this paper are: (1) a novel multimodal
system that combines muscle-sensing with interactive
surface input; (2) four proof-of-concept interaction
techniques that make use of finger identification, pressure
detection, and free-space hand movement in conjunction
with surface contact information; (3) a preliminary systemevaluation demonstrating the feasibility of our approach;and (4) a discussion of the benefits and limitations muscle

sensing offers as a complementary technology to those
employed by the tabletop community.
BACKGROUND AND RELATED WORKWe briefly review relevant work on interactive surfaces andprovide background on muscle-sensing and its use in
human-computer interaction.
Interactive Surface Sensing

While most available multi-touch systems are capable oftracking various points of user contact with a surface (e.g.,[5]), the problem of identifying particular fingers, hands, or
hand postures is less well solved. Existing approaches tosolving this problem include camera-based sensing, electrostatic coupling, and instrumented gloves.
Several camera-based interactive surface systems havedemonstrated the capability to image the user’s hands,
either above the display (e.g., [22]) or through the display(e.g., [2,20]), but none of these explore contact
identification or freehand interactions in the space above
the surface. Malik et al. [13] used two overhead cameras todetect hand postures as well as which finger of which handtouched a surface, but required a black background forreliable recognition. In general, camera-based approaches
have two shortcomings: fingers and hands can easily beoccluded and contact pressure is not robustly observable.
Techniques such as frustrated total internal reflection(FTIR) [7] are able to estimate contact pressure by
detecting changes in the shape of a contact that are often
indicative of pressure changes; however, this approach has
limited precision, and FTIR systems cannot reliably
discriminate contact shape changes due to postureadjustments from those due to pressure variation. FTIR
systems also cannot reliably identify contacts as belonging
to particular fingers.
Benko et al. [1] demonstrated a multi-finger interaction
technique which required users to wear instrumented
gloves for finger identification. Gloves have also been
extensively used in virtual reality research. For example,
Cutler et al. [4] used gloves for above-the-surface 3Dinteractions. While simple and reliable, gloves suffer frommany issues, including hygiene, comfort, access time, and areduction in the directness offered by “direct touch”
interfaces.
Interaction in the space above the interactive surface has
also been explored with styli [12], video cameras [21, 22,23], and depth-sensing cameras [2,24]. The use of depthsensing cameras is particularly of interest, as it facilitates
precise 3D hand positioning and gesture-tracking without
requiring the user to wear on-body sensors. However, lowsensing resolution, finger visibility, and occlusion issues
make such approaches potentially more error-prone than
the approach described in this paper. In addition, neither
depth-sensing or standard video cameras are able to directlysense contact pressure and require gestures to be “withinsight” of the surface. Other technologies such as Izadi etal.’s SecondLight [9] permit projection onto objects held inthe space above the surface. While supporting aninteresting set of interactions, this does not allow input
away from the surface, only output.
While not in the domain of surface computing, Sugiura andKoseki [19] demonstrated the concept of finger-dependent
user interface elements and interactions. They relied on astandalone fingerprint reader to determine which finger wasused and assigned data and specific properties to each of
the user’s fingers.
Muscle Sensing

In an independent line of work, researchers havedemonstrated the feasibility of using forearm
electromyography (EMG) to decode fine finger gestures forhuman-computer interaction [17,18]. EMG measures theelectrical signals used by the central nervous system tocommunicate motor intentions to muscles, as well as theelectrical activity associated directly with muscle
contractions. We refer the reader to [14] for a thoroughdescription of EMG.
EMG has conventionally been used in clinical settings for
gait analysis and for muscle function assessment during
rehabilitation. More recent research has explored the use of
EMG for direct input, specifically for controlling prosthetic
devices (e.g. [6,16]). Work in this area has demonstrated
experimentally that such a system can be used todifferentiate among finger and hand gestures performed byhands resting on a non-interactive table [20,25,26].
Furthermore, these gestures extend to scenarios in which
the user’s hand is not constrained to a surface, including
gestures performed when holding objects [18].
COMBINING MUSCLE AND TOUCH SENSING

Touch-sensitive surfaces and EMG provide complementarystreams of information. Touch-sensitive surfaces provide
precise location and tracking information when a user’s
hand is in contact with the surface. They can also precisely
record temporal information about the arrival and removalof contacts. EMG can detect which muscle groups, andconsequently which fingers, are engaged in the current
interaction. It can also approximate the level of activationof those muscle groups, which allows the estimation ofcontact pressure. Furthermore, EMG can provide
information about an interaction even when a user’s hand is
no longer in contact with a surface. However, EMG cannot
provide spatial information, and is not as reliable as touchsensing for temporally-sensitive gestures. We thus
introduce a multimodal system that relies on surface input
for spatial information and muscle sensing for finger
identification, pressure, and off-surface gestures.
Hardware and Setup

Our system is implemented using a Microsoft Surface(http://microsoft.com/surface) and a BioSemi Active Two
EMG device (http://biosemi.com). The EMG device
samples eight sensor channels at 2048 Hz.

training, but only a ten-second calibration procedure thatallows the system to scale pressure values appropriately.
The latency of pressure reporting is approximately 150ms.

Figure 2. EMG sensors on a user’s arm.
We placed six sensors and two ground electrodes in aroughly uniform ring around the upper forearm of theuser’s dominant hand for sensing finger gestures (Figure 2).
We also placed two sensors on the forearm of the nondominant hand for recognizing coarse muscle activation.
We chose this configuration to minimize setup complexity
while allowing us to demonstrate the feasibility ofbimanual interactions. The current asymmetric setup was aconstraint of only having 8 sensor channels, and beyond
this, more sensors on both arms would yield finer
resolution of muscles and touches.
In general, our approach was to place EMG sensors in anarrow band on the upper forearm, which we believe isrelatively unobtrusive while allowing us to sense finger
movements accurately. Our current system utilizes a wired
connection between the sensors and an amplifier, but
wireless EMG systems such as that made by ZeroWire(http://zerowire-emg.com)
have
recently
becomecommercially available. We envision our system eventually
becoming a thin wireless band worn just below the elbow.
Interpretation of Muscle Signals

Our system uses the EMG signals to provide four
primitives to applications on the interactive surface.
Level of pressure. The pressure primitive is a smoothed,down-sampled representation of the raw level of muscle
activation on the dominant hand. This feature requires noContact finger identification. This primitive is based on amachine learning methodology demonstrated in the work of
Saponas et al. [17,18]. Specifically, we use a support vector
machine to analyze frequency and amplitude information in
the EMG signal and determine which finger is applying
pressure to the surface. This primitive requires about two
minutes of training for each user. In prior work, users havetypically been asked to respond to various controlled
stimuli while the arm is in a fixed position in order tocollect labeled data. This can be tiring and boring. In our
training, we instead prompt users to use each of theirfingers to draw freely on the surface. At the end of the
training period, the system analyzes the training data tobuild a real-time classifier. Building the classifier requires
less than five seconds. The latency of finger identification
is approximately 300ms.
“Pinch” and “Throw” gestures. A “pinch” gesture consistsof bringing a finger rapidly against the thumb, and lifting
away from the surface, the way one might pick up a small
object from a table. The “throw” gesture consists of rapidly
opening the fingers from the pinched state, as one might dowhen throwing an object held between pinched fingers. The“pinch” and “throw” gestures are detected by looking forcharacteristic changes in the muscle activation level of thedominant hand. Detecting these gestures requires no
training, but identifying the fingers performing thesegestures currently requires a two-minute training procedure
identical to that described for contact finger identification,
except that instead of drawing on a surface, the system asks
the user to pinch specific fingers against his or her thumb in
mid-air for five seconds at a time during a two minutetraining period. The latency of pinch detection andidentification is also approximately 300ms.
“Flick” gesture. The “flick” gesture consists of a simple
wave of the hand. The “flick” gesture is detected bylooking for characteristic changes in the muscle activationlevel of the hand. This primitive requires no training otherthan a ten-second calibration procedure that allows thesystem to scale pressure values appropriately. The latencyof “flick” detection is approximately 50ms.
Due to the equipment constraint of having only 8 EMG
sensor channels and the resulting asymmetric setup, webound each of the gestures to a specific hand. The
dominant hand, with the larger number of sensors, could
sense pressure, contact-finger identification, as well as thepinch and throw gestures. The flick gesture was restricted
to the non-dominant hand.

Figure 3. An example drawing demonstrates bothpressure-painting and finger-dependent painting. A
different color is mapped to each finger, andpressure controls stroke saturation.

Calibrating and training our system for all four primitives
requires approximately five minutes per user. We discussincorporating these primitives into hybrid interaction
techniques in the next section.

cut photo by maintaining the pinch posture, even after thehand has left the surface, and pastes the object back onto
the surface by executing the throw gesture. The user canperform arbitrary actions (e.g., switch between canvases)
while she is holding the object and has not thrown it back.
Undo flick: To demonstrate our system’s ability tofacilitate bimanual, off-the-surface interaction, we map the“flick” gesture performed by the non-dominant hand to theundo operation in our painting application. This action
removes the most-recently-created stroke.
EXPLORATORY SYSTEM EVALUATION

Figure 4. Performing the finger-dependent pick andthrow interaction: A user picks up a virtual object
by pinching it on the surface and lifting his handaway from the surface. Releasing the pinch return
the object the current canvas.

To gather initial feedback on our system, we recruited 6
participants (3 female) from within our organization. Eachparticipant spent approximately 90 minutes interacting with
our system and was provided with a $10 compensation for
their time. The goals of our evaluation were to validate thebasic feasibility of our system and interaction techniques,to assess their robustness and reliability, and to gather
anecdotal responses from novice users about our proposedinteraction techniques.

Hybrid EMG-Surface Interactions

Tasks

We have prototyped four interaction techniques todemonstrate and evaluate the utility of EMG sensing forinteractive surfaces. These interactions are all prototyped
within a simple painting and image-manipulation
application.

At the beginning of each participant’s experimental session,
we applied EMG sensors to the participant’s arm as
described in the previous section. We then asked theparticipant to make a tight fist and then relax, allowing
calibration of the signal level for each hand. Introduction
and the initial setup took approximately 15 minutes.
Participants then completed the following five tasks (inorder):

Pressure-sensitive painting: To demonstrate our system’sability to estimate contact pressure, we associate different
saturation levels in our painting application with different
levels of finger pressure (more pressure results in darker
strokes) (Figure 3).
Finger-aware painting: To demonstrate our system’sability to associate surface contacts with specific fingers,
we associate different brush colors with the index andmiddle fingers (Figure 3). When the interactive surface
detects a contact, it immediately queries the EMG system
for the identity of the active finger, and uses that color forthe brush stroke associated with this contact. Because wehave independent processing streams for touch and muscle
sensing, we begin to draw a translucent stroke to maintainthe sensation of responsiveness, and only fill in the color
when the EMG system has returned with the finger it
detects.
Finger-dependent pick and throw: To demonstrate oursystem’s ability to detect gestures more complex thansimple touches, and to persist the state of those gestureseven when the hand leaves the surface, we map the “pinch”
and “throw” gesture primitives to “cut/copy” and “paste”
operations on a simple photo canvas. Thus the user is able
to pick a photo up from the table and throw it back onto thecanvas. Picking is initiated on the surface, by placing two
fingers on the desired photo and then performing a pinch
gesture (Figure 4). By pinching with the index or middle
finger, the user can specify whether to initiate a cut or a
copy operation, respectively. A user holds on to a copied or

Task 1: Copy an image from a given paper template(Figure 5a) using the pressure-sensitive painting technique.
The image was presented on paper and contained varying

Figure 5. Four tasks from our user evaluation: (a)Task 1: copy an image using contact pressure tocontrol saturation; (b) Task 2: copy an image using
index and middle fingers to paint two separate
colors; (c) Task 3: draw lines with alternating colors;and (d) Task 5: move three images and copy three
images to a different canvas.

levels of light and dark strokes.
Task 2: Copy an image from a given paper template(Figure 5b) using the finger-aware painting technique. The
image was presented on paper and contained blue andgreen strokes, which were mapped to the participant’s
index and middle fingers, respectively.
Task 3: Make a series of vertical lines across the surface,changing color with each vertical line (Figure 5c). Eachparticipant filled two canvases with vertical lines.
Task 4: Write the numbers from 1 to 10 on the surface,executing the “undo flick” gesture after each even number,but not after odd numbers. Correct execution of this task
would leave only the odd numbers written on the surface. If
an even number contained multiple strokes, participants
executed the “undo flick” gesture as many times as wasnecessary to erase the number completely.
Task 5: Presented with a pile of six images on a canvas,
either copy or move each image to another canvas,
depending on the image category. Specifically, they had tocopy images of cats and move images of dogs (Figure 5d).
Participants picked up images using our “pick” gesture,
where the index finger initiated a “move/cut” operation andthe middle finger initiated a “copy” operation. While theimage was held in their dominant hand, participants pressedan on-screen button with their non-dominant hand to switch
to the target canvas, and used the “throw” gesture to place
the image on that canvas.
There were two additional training sessions: First, before
performing Task 2, participants spent two minutes training
the system to recognize finger-specific contacts. Second,participants spent another two minutes training the fingerspecific “pinch” gesture before Task 5. Training in bothcases consisted of repeated activation of a desired handpose or gesture using a stimulus-response training method,i.e., the user was prompted with a particular pose/gestureon the screen, they performed it for 2 seconds, and thenthey relaxed their hand muscles.
Before performing each task, participants were given timeto practice each interaction and ask questions. This practicesession took no longer than 5 minutes. When comfortablewith the interaction, participants proceeded to complete thespecific tasks, which were untimed. On average,
participants completed each task within one minute.
At the conclusion of the session, each participant
completed a questionnaire that solicited feedback about
each interaction.
Results

In this section, we present quantitative results from each of
our tasks. Discussion of the implications of these results ispresented in the following section.
Task 1: We analyzed Task 1 (copying an image usingpressure-sensitive painting) by defining 22 features, such as“line 2 is lighter than line 1”, and “line 3 demonstrates thecorrect brightness gradient”, and coding errors on each ofthese features for each participant. The resulting drawings
can be seen in the top row of Figure 6. Across our six
participants, the mean accuracy was 93.9% (sd = 4.7%). In
short, all participants were able to effectively manipulatepressure to control brush darkness in a drawing task.
Task 2: The task of copying a multi-color image is more
open-ended and therefore difficult to formally analyze, as
participants used different numbers of strokes to complete
the image. Anecdotally, success on task 3 (vertical lines)
was indicative of users’ ability to perform task 2: while all
six participants completed the target drawing (middle rowof Figure 6), one had some difficulty reliably selecting thefinger color.
Task 3: We analyzed Task 3 (finger-aware drawing ofalternating blue and green vertical lines) by computing thepercentage of lines drawn in the correct color for eachparticipant (see bottom row of Figure 6). Across our six
participants, the mean accuracy was 90.9% (sd = 11.1%).
This includes one participant for whom finger classification
did not perform at a level comparable to the otherparticipants. In this errant case, the classification wasbiased toward one finger, resulting in an accuracy of only
71%. Without this participant, the mean accuracy overall
was 94.8%. In short, five out of six participants were able
to effectively specify brush colors by painting with different
fingers.
Task 4: We analyzed Task 4 (writing numbers andselectively erasing half of them with the “undo flick”
gesture) by counting the number of false-positive and falsenegative “undo” operations performed by each participant.
All participants but one completed this task with no errors.
The one participant had two false positive errors. In short,five out of six participants were able to reliably execute andcontrol the “undo flick” gesture without any false positives.
Task 5: We analyzed Task 5 (picking and throwing
images) by counting the number of “mis-triggers” and“mis-classifications.” Mis-triggers were instances where
the system detected a pinch or throw gesture that the user
did not intend, or failed to detect an intended gesture. Misclassifications were instances where the system correctly
detected the presence of a “pick” gesture but failed tocorrectly identify the gesturing finger. Three of our six
participants performed this task without any errors of either
type. Two of the remaining three participants experienced
no mis-triggers, but had 2 and 3 mis-classifications,
respectively. The remaining participant experienced 2 mistriggers and 1 mis-classification. In short, this was the mostdifficult of our interactions, but the three perfect executionsof this task support its basic feasibility. In the following
section, we will discuss hypotheses surrounding theclassification errors experienced by the other participants.
In summary, while it is important to keep in mind that webase our observations on a very limited set of six
participants, only one experienced difficulties getting
reliable recognition, while five performed all tasks without
problems.

Figure 6. Pictures painted by participants in our exploratory system evaluation, where rows 1, 2, and 3 show theresults of Tasks 1, 2, and 3 respectively. Task 1: copy the leftmost image using pressure-sensitive painting. Task 2:
copy the leftmost image using index and middle fingers to paint in blue and green, respectively. Task 3: draw
alternating blue and green lines using index and middle fingers, similar to task 2. The leftmost target images were
provided to our participants on paper.
DISCUSSION AND FUTURE WORK

Here we discuss the lessons learned from developing oursystem and testing our interaction techniques, and present
opportunities for future work.
Calibration and Training

One goal when developing sensing and recognition systems
is to construct an accurate model that requires minimal
calibration and training. Our system currently requires
gross calibration each time a user dons the EMG device.
This comes in the form of a making a tight fist and thenrelaxing each of the hands. Because of the variance inmuscle activity across users and the inconsistency in sensor
placement, even for repeated use on the same user, this isnecessary to normalize the raw amplitudes and find thebasic working range of the signal. This calibration provides
sufficient information to model pressure gestures, pick andthrow gestures, as well as our flick gesture, since thesefunction based on thresholds set on the signal amplitude.
Other gestures such as distinguishing between different
fingers require more training since the relationship between
the raw signal and the desired recognition result is lessobvious. In these cases, we have users perform tasks inwhich we collect labeled data that can be used by machinelearning techniques to dynamically build the classification
model.
We believe that this training exercise must be carefully
designed in order to collect data that is representative ofreal use scenarios. For example, traditional EMG training
methodologies have largely employed a stimulus-response
paradigm, in which the user is told exactly which gesture to
perform and when. In the case of finger identification, we
could have had the user press down with each of theirfingers when we told them to. This is not only potentiallyboring and annoying to perform, but also provides data thatis quite different from that which has to be recognized.
In our tests, we had users paint images of their choice while
using fingers of our specification, which was a much morecompelling exercise that provided better training data. Eventhen, some users performed the training very differently
than they did the task. While we cannot quantify this, ourinformal observations of the user that had poor recognition
results leads us to believe that they were trying so hard to
train the system correctly that their arm might have beenabnormally tense when they did this, leading to theconstruction of a poor model.
These issues point toward a limitation of our current
system. We did not explicitly tell users that they had toperform the tasks and gestures in any given way, and we
found that users who deviated most from the way theytrained the system generally had the worst recognition
results. This is hardly surprising, but most users were able
to naturally self-correct after the training phase, and with afew minutes of practice, quickly learned how to performthe gestures in a way as to get reliable classification.
Classification Limitations

The recognition rates achieved by our system – for example
the 90% mean accuracy for finger identification – might beconsidered low when compared to the error rates ofstandard input devices such as mice and keyboards.
However, our accuracies are comparable to other “nontraditional” input modalities such as speech and gesture,both of which have achieved success in a variety of
applications where the benefits of alternative modalities
compensate for reduced accuracy. In addition, we believethat a more synergistic combination of touch sensing andmuscle sensing would probably yield better recognition
results. For example, we could consider the changes in thetouch contact area as well as the outline of the hand in thehover zone to further aid our recognition system.
The present work also did not explore the long-term
performance of our classifiers, and finding techniques thatcreate models that are robust to variations in sensor
placement and user performance remains future work. The
need to individually place electrodes on a user’s arm limits
the reusability of training data and thus the long-termrobustness of our system, but we are currently investigating
a novel, dry-electrode armband form factor which allows
the user to quickly attach sensors to their arm. Thisapproach shows potential in facilitating calibration datareuse across multiple sessions for each individual user.
Gesture Sets

In this work, we only classify a single contact at a time.
This is not an intrinsic limitation of the approach, but rather
one of implementation. It remains future work to develop
recognition techniques that deal with compound gestures,
whether through training explicitly for these gestures or byinferring them from models of the individual gestures. If
multiple digits are touching the surface at the same time,
the system could also use the relative position and orderingof the fingers and the information about which fingers are
currently touching the surface to infer which finger iswhich. Even minute changes in pressure and finger flex
could be correlated with minute changes in finger contact
area and relative position to other contacts to precisely
identify each finger in contact with the surface.
One of our explicit design decisions was to utilize only theindex and middle fingers. This was a simplification since
we sought to explore modality fusion rather than explicit
EMG system performance. That said, [17] and [18] haveshown that the recognition accuracy does not degrade
drastically even when people use all five fingers. This work
demonstrated that the little finger was the least reliable for
EMG classification, which we believe is acceptable since
the little finger is typically the least comfortable to design
gestures around. It should be noted that we expect that thenatural way to use the thumb on a surface is probably not
equivalent to the best-case scenario tested in that work andwe would likely see slightly degraded performance there as
well; the muscles controlling the thumb, are less accessible
to a forearm EMG sensor than the muscles that drive theother fingers.
Interaction and Interface Considerations

A slightly more opportunistic idea is to make use of aunique property of muscle-sensing: it is sometimes possible
to detect a physical movement event before it actuallyoccurs. This is because before we make a motion, we have
preparatory muscle activation that can be sensed by EMG.
Hence, it may be possible to detect actions such as pressing
a button slightly before the physical event actually occurs,
which could perhaps be integrated into interaction
techniques for tabletops to, for example, begin animating achange to an object that will be affected on screen.

Figure 7. Finger-dependent UI elements: (a) finger
ink wells for choosing the brush color of index andmiddle fingers, and (b) middle-finger quit button to
reduce accidental activation.
In our prototype system, we implemented and evaluatedeach of our interaction techniques separately. However,these can obviously be integrated into a single system.
Pressure sensing can be done simultaneous with finger
identification and the surfaces ability to sense contact shapefor hybrid interactions such as simultaneously controlling,
stroke shape, color, and saturation. Similarly, finger
identification on the surface (e.g., painting) and finger
identification off the surface (e.g., pinching) can be inferred
simultaneously through separate classifiers while using
surface contact information to determine how to use theresults.
In addition to our hybrid interaction techniques, we
explored the concept of finger-dependent user interface
elements (Figure 7), i.e., on-screen elements that can be
activated only when touched with a specific finger (similar
to the concept introduced in [19]). We prototyped fingerdependent ink-wells for selecting the finger brush color,and middle-finger quit button for exiting our application.
Such elements are harder to activate by mistake thanstandard widgets, which could be useful for actions withhigh cost of accidental activation (e.g., delete or quit).
CONCLUSION

We have presented a novel fusion of complementarysensing modalities: touch sensing via an interactive surface
and muscle sensing via EMG. Our approach enhances theexisting tabletop paradigm and enables new interaction
techniques not typically possible with standard interactive
surfaces. Our exploratory system evaluation provides
important insights into the feasibility, reliability andeffectiveness of our approach. We believe that with the
future development of miniaturized, wireless, and wearable
EMG sensing devices, our techniques will provide usefulinteraction capabilities for the next generation of interactive
surfaces.
REFERENCES

1. Benko, H. & Feiner, S. Balloon Selection: A Multifinger Technique for Accurate Low-fatigue 3D
Selections. In Proc. of Symp. on 3D User Interfaces „07.
79–86. 2007.
2. Benko, H. & Wilson, A. DepthTouch: Using DepthSensing Camera to Enable Freehand Interactions On
and Above the Interactive Surface. Microsoft ResearchTechnical Report MSR-TR-2009-23. 2009

3. Costanza, E., Inverso, S.A., Allen, R., & Maes, P.
Intimate Interfaces in Action: Assessing the Usability
and Subtlety of EMG-based Motionless Gestures. In
Proc. of ACM CHI 2007.
4. Cutler, L.D., Fröhlich, B., & Hanrahan, P. Two-handed
Direct Manipulation on the Responsive Workbench. In
Proc. of Symp. on Interactive 3D Graphics (I3D ‟92).
107-114, 1992.
5. Dietz, P. & Leigh, D. DiamondTouch: A Multi-user
Touch Technology. In Proc. of ACM UIST ‟01. 219–
226, 2001.
6. Farry K., Walker I., & Baraniuk R.G. Myoelectric
Teleoperation of a Complex Robotic Hand. In Proc.ofIEEE Intl Conf Robot Automation ’96. 775-788, 1996.
7. Han, J. Low-cost Multi-touch Sensing throughFrustrated Total Internal Reflection. In Proc. of ACM
UIST ‟05, 115–118, 2005.
8. Harada, S., Saponas, T.S., & Landay, J.A. VoicePen:
Augmenting Pen Input with Simultaneous Nonlinguistic Vocalization. In Proc. of Intl. Conf. on
Multimodal Interfaces, 2007.
9. Izadi, S., Hodges, S., Taylor, S., Rosenfeld, D., Villar,N., Butler, A., & Westhues, J. Going Beyond the
Display: A Surface Technology with an Electronically
Switchable Diffuser. In Proc. of ACM UIST ‟08, 269–
278, 2008.
10. Ju, P., Kaelbling, L.P., & Singer, Y. State-based
ClassificationofFingerGesturesfromElectromyographic Signals. In Proc. of ICML ‟00, 439446, 2000.
11. Julia, L. & Faure, C. Pattern Recognition andBeautification for a Pen-based Interface. In Proc. of Intl.
Conf. on Document Analysis and Recognition (Vol. 1),
58, 1995.
12. Kattinakere, R.S., Grossman, T., & Subramanian, S.
Modeling Steering within Above-the-surface Interaction
layers. In Proc. of ACM CHI ‟07, 317-326, 2007.
13. Malik, S., Ranjan, A., & Balakrishnan, R. Interactingwith Large Displays from a Distance with Visiontracked Multi-finger Gestural Input. In Proc. of ACM
UIST ‟05, 43–52, 2005.
14. Merletti, R. & Parker, P.A. Electromyography:
Physiology,
Engineering,andNoninvasive
Applications. John Wiley & Sons: Hoboken, New
Jersey, 2004.

15. Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L.,
Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay,
J. A., Larson, J., & Ferro, D. Designing the User
Interface for Multimodal Speech and Pen-based Gesture
Applications: State-of-the-art Systems and Future
Research Directions. HCI 15, 263–322, 2000.
16. Peleg, D., Braiman, E., Yom-Tov, E., & Inbar G.F.
Classification of Finger Activation for Use in a Robotic
Prosthesis Arm. Trans. Neural Syst. Rehabil Eng.,
10(4), 2002.
17. Saponas, T. S., Tan, D. S., Morris, D. & Balakrishnan,
R. Demonstrating the Feasibility of Using Forearm
Electromyography for Muscle-Computer Interfaces. In
Proc. of ACM CHI ‟08, 515–524, 2008.
18. Saponas, T. S., Tan, D. S., Morris, D, Balakrishnan, R.,
Landay, J.A., & Turner, J. Enabling Always-available
Input with Muscle-Computer Interfaces. In Proc. of
ACM UIST 2009.
19. Sugiura, A. and Koseki, Y. A user interface using
fingerprint recognition: holding commands and data
objects on fingers. In Proc. of ACM UIST '98, 71–79,
1998.
20. Tenore, F., Ramos, A., Fahmy, A., Acharya, S.,
Etienne-Cummings, R., & Thakor, N. Towards theControl of Individual Fingers of a Prosthetic Hand
Using Surface EMG Signals. In Proc. of IEEE EMBS
2007.
21. Wilson, A. TouchLight: An Imaging Touch Screen and
Display for Gesture-based Interaction. In Proc. of ICMI
„04. 69–76, 2004.
22. Wilson, A. PlayAnywhere: A Compact Tabletop
Computer Vision System. In Proc. of ACM UIST ‟05.
83–92, 2005.
23. Wilson, A. Robust Computer Vision-Based Detection of
Pinching for One and Two-Handed Gesture Input. In
Proc. of ACM UIST ‟06. 255-258., 2006.
24. Wilson, A. Depth-Sensing Video Cameras for 3D
Tangible Tabletop Interaction. In Proc. of IEEE
Tabletop ‟07, 201-204, 2007.
25. Wheeler, K.R, Chang M.H., & Knuth K.H. GestureBased Control and EMG Decomposition. IEEE Transon Systems, Man, and Cybernetics, 36(4), 2006.
26. Yatsenko, D., McDonnall D., & Guillory, S.
Simultaneous, Proportional, Multi-axis Prosthesis
Control using Multichannel Surface EMG. In Proc.
IEEE EMBS 2007.