Pre-Touch Sensing for Mobile Interaction
Ken Hinckley1, Seongkook Heo1,2, Michel Pahud1, Christian Holz1, Hrvoje Benko1, Abigail Sellen3,
Richard Banks3, Kenton O’Hara3, Gavin Smyth3, and Bill Buxton1,3
1
Microsoft Research, Redmond, WA, United States, {kenh, mpahud, cholz, benko}@microsoft.com
2
HCI Lab, Department of Computer Science, KAIST, Republic of Korea, seongkook@kaist.ac.kr
3
Microsoft Research, Cambridge, UK, {asellen, rbanks, keohar, gavin.smyth, bibuxton}@microsoft.com
ABSTRACT

trajectory indicates likely targets [60,61]. This treasury ofcontextual detail—which we collectively refer to as pretouch (a term previously used by [1,49])—is lost to mobile.

Touchscreens continue to advance—including progress
towards sensing fingers proximal to the display. We explore
this emerging pre-touch modality via a self-capacitancetouchscreen that can sense multiple fingers above a mobile
device, as well as grip around the screen’s edges. This
capability opens up many possibilities for mobile interaction.
For example, using pre-touch in an anticipatory role affords
an “ad-lib interface” that fades in a different UI—appropriate
to the context—as the user approaches one-handed with athumb, two-handed with an index finger, or even with apinch or two thumbs. Or we can interpret pre-touch in a
retroactive manner that leverages the approach trajectory todiscern whether the user made contact with a ballistic vs. afinely-targeted motion. Pre-touch also enables hybrid touch+ hover gestures, such as selecting an icon with the thumbwhile bringing a second finger into range to invoke a context
menu at a convenient location. Collectively these techniques
illustrate how pre-touch sensing offers an intriguing new
back-channel for mobile interaction.

Figure 1. The pre-touch sensing modality detects multiple
fingers above and around (gripping) the edges of the screen.

Pre-touch sensing above and around the screen (Fig. 1)
therefore lends new insights to mobile interaction. We focus
on contextually rich aspects of touch that take place before,
or in conjunction with, actual contact—as opposed toaftertouch, a term for pressure-sensitive response [14,63] aswell as in-air suffixes for gestures [9,33]. The richness of the
pre-touch modality, which encompasses both grip sensing
and multi-finger proximity, also distinguishes it from hover[12,20,33], which connotes a discrete state for tracking asingle point (cursor) on legacy input devices [8]; or in-air
gestures [9,25,50], which focus almost exclusively on overt
actions. By contrast, our work on pre-touch emphasizes morecasual, adroit, and context-driven interpretations [7,27,45].

Author Keywords

Multi-touch; hover; grip; context sensing; mobile interaction
ACM Classification Keywords

H.5.2 [Information Interfaces and Presentation]: Input
INTRODUCTION

Natural human grasping behavior is analog and continuous.
Yet the touchscreen of a mobile device typically restricts
designers to flatland: a world of discrete state transitions
defined by an impoverished, on-screen, two-dimensionalview of the human hand.

Our resulting techniques therefore illustrate three strategies
for pre-touch sensing in interaction design:

The problem is that much of what characterizes ‘touch’ starts
before contact [16] and originates from beyond the confinesof the screen. Users first grip their mobile with the left handor the right [18,22,57]. They then reach for the screen withan index finger, one-handed with a thumb [4,30], or withmultiple digits for pinch-to-zoom. As the hand approaches,the posture hints at the user’s intent [13,37,44] and the

Anticipatory reactions modify the interface based on theapproach of the fingers, in a manner that furthermore may becontingent on grip. For example, we demonstrate a mobile
video player with an ad-lib interface that fades in when theuser’s fingers approach the screen, and fades out when theuser moves away. These controls are context-sensitive: theirpresentation depends on the current grip, which direction thehand approaches from, and the number of fingers.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from Permissions@acm.org.
CHI'16, May 07 - 12, 2016, San Jose, CA, USA
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-3362-7/16/05…$15.00
DOI: http://dx.doi.org/10.1145/2858036.2858095

Retroactive interpretations construe touch events based on
how the user approached the screen. We show techniquesthat reinterpret tap or drag events based on whether the user
approached the screen in a ballistic motion, or with a finelyadjusted trajectory, allowing on-contact discrimination
between flick-to-scroll vs. text selection, for example.
1

Yet how we think about these sensors—forsaking the lowhanging fruit of new ways to signal overt gestures in favor of
the less obvious, more contextual ways to use emerging
modalities—may be a key pivot in our perspective.

Hybrid touch + hover gestures combine on-screen touchwith above-screen aspects, such as selecting an object withthe thumb while bringing the index finger into range to callup a Hybrid Menu. This reveals contextual options without
resort to a time-out, in an easy-to-reach position. Althoughan overt use of pre-touch, this represents an under-exploredclass of hybrid gesture—in a way that also uses grip sensing“in the background” of the interaction [7] to support gracefuldegradation to a one-handed version of the technique.

Foreground vs. Background Interaction

Buxton introduces a simple model of foreground versus
background interaction [7]. The foreground includes activity
that is at the fore of the user’s attention, such as flipping alight switch at the entrance to a room—or tapping a target ona mobile touchscreen. But the background characterizes thecontext of activity taking place ‘behind’ the foreground—such as sensing the user walk into a room, and turning on thelights in response—or sensing the user’s fingers approach thescreen, and fading in a context-appropriate interface to suit.

In sum, then, our work contributes the following: The first exploration of pre-touch on a fully mobile device,
particularly with regards to background-sensing aspects;
 Mobile interaction techniques that combine rich abovescreen proximity with around-screen grip for some (butnot all) of the design strategies we identify for pre-touch: anticipatory reactions that adapt a mobile interface
based on the context revealed by pre-touch; retroactive interpretations that augment touch eventswith the trajectory of the approaching finger(s); and hybrid touch + hover gestures.
 A design space organizing these key aspects of pre-touch; And preliminary user feedback on our techniques.

Common Problems in Mobile Interaction as Missing Context

This perspective also helps us to see how many common
problems in mobile interaction—such as one-handed
interaction [4,30], occlusion of the screen by the fingers, oreven the fat finger problem—might be re-framed asproblems of context. Sensing which hand is holding thedevice fosters appropriate one-handed adaptations [18,57].
Occlusion can be avoided if the device can infer what content
the hand is blocking [22,55,57]. Fat fingers can be partiallyremedied by sensing the posture of the touch [28]. And soforth. In this paper we explore just a few techniquesmotivated by some of these problems, but if pre-touch
becomes commonplace it may prove useful in attacking theseand many other difficulties in mobile touch interaction.

Collectively these contributions illustrate the promise of pretouch as a sensing modality, and point the way to the stilllargely untapped potential of ‘touch’ once we free ourselvesfrom the flatland of the standard touchscreen.
RELATED WORK

Our work relates insights from human grasping behavior toa viewpoint informed by sensing techniques, and the lens ofbackground sensing in particular, to re-frame some common
issues in mobile interaction as problems of context. Both grip
and in-air (hover) sensing are key to realizing this direction.

Grip Sensing Techniques for Mobiles

Grip sensing can enrich mobile interfaces in many ways. For
example, grip sensors can determine whether the user isholding a device in the left hand or the right [18,22,57],
automatically bring up a viewfinder when the user holds thephone like a camera [32], or suppress automatic screen
rotation when the user’s grip remains unchanged [10]. Other
work has shown that grip, or the change in grip implied bymotion sensors [42], can be used to anticipate the generalarea where the user is about to touch [43]. Grip sensors canalso adapt the interface to suit the context [54], such as bybringing up a graphical keyboard at a convenient location[11]. Our work advances this theme in a nuanced way that
fades in or fades out multiple, contextually-appropriate
variations of a mobile user interface in an ad-lib fashion.

Natural Human Grasping Behaviors

During prehension, the hand shapes itself—even prior tocontact—to grasp tools for a specific purpose [37]. This isreflected by the posture of the hand as well as the kinematicsof the reaching movement itself [13,39,44]. For example,
probabilistic pointing [19] and expanding widgets [40,61]
leverage the two-phase nature of pointing movements: rapid
ballistic motion is followed by fine adjustment [41]. We use
this insight by reasoning that trajectories with a distinct fineadjust phase are likely intended for small targets (not large).

Sensing Hover and In-Air Interactions

At a higher level, in human skilled bimanual action—such as
pointing with one hand at a phone held in the other—thenonpreferred hand (grip) precedes and sets the frame ofreference for the activity of the preferred hand [21]. Thus
mobile interaction (at least in its two-handed manifestation)
is a compound task that involves both hands in contact withthe device, even if only the contribution of the preferred hand
has traditionally been deemed as a ‘touch.’

A hover state for touch has recently started to appear on somemobiles—albeit typically restricted to single-finger hover—
but the modality has long been explored on larger formfactors such as tabletops. The Continuous Interaction Space[38], perhaps the first work to explicitly recognize thecontinuity between hover and on-screen touch, explores
interactions such as providing feedback of possible actions.
Our approach to pre-touch builds on this continuity, unifies
it with grip sensing, and advances it for mobile interaction.

Sensing Techniques

User experiences with technology are increasingly mediatedby sensors [3]. In the context of mobile computing, researchon hover, as well as grip and motion, is particularly relevant.

A proposed model of feed-forward for multi-touch [16]resonates with our insight that grip informs the pre-input2

we call out the ground—that is, the use of fore- vs.
background sensing [7], with the background shown in gray.

phases of touch. Medusa [1] also employs pre-touch
feedback by sensing users approaching a tabletop display,with “Just-in-Time Widgets” that appear when users hold an
arm above the tabletop. By contrast, our ad-lib interface
appears when fingers naturally approach a mobile device:indeed, numerous context-appropriate versions of our UI,
which are contingent on both grip and hover, fade in or fadeout to accommodate various aspects of mobile interaction,
thus going well beyond previous work [1,12,16,60].

The columns encompass our anticipatory, retroactive, andhybrid design strategies. Although these focus primarily on
temporal aspects of the interactions, as design categories
they admittedly lack absolutely rigid demarcations—andadditional general strategies could be devised, as well. In thisspirit, note that one could potentially add aftertouch—either
by considering it as another strategy (e.g. for in-air suffixes
to touch [9,33]), or by treating pressure [14,23,24] as aproperty sensed (e.g. Apple 3D touch [63])—in future work.

HOVER
GRIP
GRIP + HOVER

Back-

Proposed: extension of Hybrid Menu: grip
Ballistic vs. Fine Tap
triggers graceful
to take into accountdegradation to onegrip as well (seehanded menu
Informal Evaluation)

Fore-

Ad-Lib Interface:controls fade in
depending on thecurrent grip

Back-

Another intriguing background use is to consider the motion
trajectory itself. Zero-Latency Tapping [60] eliminates
perceptible latency on a tabletop display by presenting ‘soft
feedback’ in anticipation of the user’s predicted landing
point. TouchCuts and TouchZoom [61] explore a directtouch variant of Expanding Widgets [40] that expands icons
based on the user’s predicted touch-down location. Our
techniques instead focus on mobile interaction, and consider
both grip and multi-finger hover as context.

Grip activates camera Predict touch from back Detecting Unintentional[32], shifts margin forgrip [43], Grip change Thumb Contact [26]annotations [22]as side-channel [42]iGrasp keyboard [11]Grip+micro-mobility [62]iRotate grasp [10]ContextType [17]Hybrid Menu: thumbselects, finger inrange calls up menu

Fore-

Calm Web Browser:
Ballistic vs. Fine Tapfeather in links; multi- Flick vs. Selectfinger gesture guides
Occlusion-aware menu Paperweight metaphor PinchPad [58][5][48]
Grip micro-interactions
as overt gesture [59]Background

While foreground uses of hover tend to dominate theliterature, examples of background uses do exist. For
example, sensed hand shadows can enrich telepresence byshowing communicative gestures in reference to a sharedtask space [16,52,53]. The imprecision of in-air interaction
lends itself to more casual interaction at the periphery ofattention [45]. And in addition to Medusa [1], other designs
suggest controls that appear on approach [12], and dissolve
when the finger moves away [60].

Anticipatory
Retroactive
Hybrid with Touch
ReactionInterpretation(on touchscreen)
Air + Touch [9]HoverWidgets [20]AnglePose [46]: adds
Continuous Interaction Gesture continuations
finger pose to touch
Space [38](Air + Touch [9],Anchored above-screen
Expanding Widgets [40] HoverFlow [33])
3D interaction [29]Sony [51], Samsung
Galaxy 4 [47]Zero-latency tap [60]Casual interactions [45] FlexAura [35] (pen withHand shadows [52,53] Palm rejection [2]IR proximity-based
Fade in [12] / out [60]sensing for handTouchCuts / Zoom [61]posture recognition)

Foreground

Air+Touch [9] explores a vocabulary of in-air gestures that
occur before, between, or after touches. While this opens up
a rich design space of overt (foreground) gestures, our work
adopts a complementary viewpoint that primarily considers
how the proximity of multiple fingers, and grip, can serve asa background-sensing modality in support of mobileinteraction. Our work also considers hybrid gestures withsimultaneous touch and hover. This possibility has only been
hinted at by one previous example, which uses “anchored”
interactions with nonpreferred-hand touch to enable in-air
gesture with the preferred hand for 3D interaction [29].

Figure 2. Design space of pre-touch, with rows for grip, hover,and grip+hover from the perspective of foreground vs.
background interaction—and columns for our three strategies
for leveraging pre-touch sensing in interaction design.

While many foreground techniques for both hover and grip
have been proposed, here we have intentionally emphasized
examples of background sensing, since those are the most
relevant to the ideas developed in this paper. The design
space also underscores that (to our knowledge) grip + hover
have not been used together before. Thus many (but not all)of the following techniques seek to explore this combination.

Summary

Our contribution not only conceptually unifies grip andhover sensing under the umbrella of pre-touch, but alsooffers an interesting application of the background sensing
point-of-view to these modalities. Even in cases where we do
propose overt gestures—such as our hybrid touch+hovergestures—this consideration led us to bolster them withbackground attributes—such as accommodating gracefuldegradation to a one-handed version of the technique.

PRE-TOUCH HARDWARE

The device that we employ is an engineering prototype of aself-contained mobile device based on the Fogale Sensation[15] technology, which uses a self-capacitance touchscreenwith a 16x9 matrix of sensors. Hence it is merely an enabling
third-party technology: we do not claim it as a contribution.
It looks and feels like a normal smartphone, weighing 175 g
and measuring 142 x 74 mm, with a maximum thickness of
12.5 mm which tapers to 7.5 mm at the radiused edges.

A DESIGN SPACE OF PRE-TOUCH INTERACTIONS

We devised a design space (Fig. 2) to situate our techniques(shown in bold) in relation to previous hover and grip-based
interactions, suggest connections between techniques, anddirect attention to relatively under-explored combinations.
On the left side, the rows indicate the property sensed: hoveror grip, plus their use in tandem (grip+hover). On the right,
3

fingertip blobs; the aspect ratio indicates whether the finger
is upright or oblique. We estimate yaw angle by finding theangle with the least brightness change along the fingertip
blob. We then combine these metrics to determine if the blobis most likely a thumb (of the hand holding the device) or afinger from the other hand: if it is oblique and came from thesame side that the user is gripping, it is thumb. Otherwise it
is a fingertip. This heuristic works well for our purposes.

The 5.2” touchscreen (16:9 aspect) senses 14-bit capacitancefor each cell of the matrix, with a 120 Hz sampling rate. The
presence of a fingertip can be sensed approximately 35 mm
above the screen, but the range depends on total capacitance(e.g. a flat palm can be sensed ~5 cm away). Thus thecapacitance values are a proxy—but not a direct measure—of distance. Grip can only be sensed close to the edges (Fig.
3); fingers on the back side of the device cannot be detected.

INTERACTION TECHNIQUES

We explore interaction techniques for each of the
anticipatory, retroactive, and hybrid touch+hover design
strategies that we identified. As such these are not cut anddried categories, but rather a palette of approaches that canbe mixed and matched—or supplemented by new strategies
in the future. While we do consider some ways to use grip
and hover for overt interaction, we pay particular attention totechniques that employ these 3 strategies in the background.
Figure 3. Hardware response to hand grip, with (a) raw 16x9
sensor image and (b) our resulting interpolated image.

ANTICIPATORY REACTIONS TO PRE-TOUCH

Anticipatory techniques proactively adapt the interface to thecurrent grip and the approach of the fingers. That is, as one
or more fingers enter proximity, the system uses the current
grip, the number of fingers, and the approach trajectory topresent an appropriate interface—or to otherwise adapt thegraphical feedback to suit the shifting context of interaction.

IMAGE PROCESSING

We implemented our algorithms in C# and C++ using the
OpenCV image processing library. Processing requires~35ms per frame. For rapid prototyping purposes, wewirelessly transmit the 16x9 matrix of sensor values to a PC,
process the image, and send the results back.

Ad-Lib Interface Controls: A Mobile Video Player

Our video player uses ad-lib interface controls (Fig. 5) topresent interactive elements in an appropriate manner, at an
appropriate location—and only when they are needed. Whena finger approaches, the system senses this and responds sothat the interface can appear “just in the nick of time.” Wepursued a video player because consuming videos on amobile device exhibits many challenges typical of mobile
applications: the more casual interaction context, the need toconsume content from a variety of grips (including onehanded), and the desire to have a minimal default interface.

Fingertip Extraction

Some of our techniques use the trajectory of an approachingfinger, and hence must identify the fingertip in the image. As
illustrated in Figure 4, which provides an example of a single
thumb approaching the screen, we follow a five-step pipelineto achieve this. We take the raw 16x9 image (Figure 4a) andinterpolate it to 180x320 using the Lanzos 4 algorithm (b). A
first fixed threshold removes the background noise of thecapacitance sensor (c). We then increase contrast and apply
a second threshold to isolate the fingertip region (d).

Related Approaches

A few prior examples have hinted at some aspects of thisapproach. For example, a previous design concept proposesthat video controls could fade in with the approach of a finger[12], as does just-in-time chrome [56]. Zero-latency tapping
suggests a complementary idea, that controls could fade out
when the finger lifts, as one of its proposals for future
directions [60]. And the Medusa tabletop [1] fades in certain
controls at the approach of a hand, or fades in gesture guides
when the hand hesitates at the periphery of the display.

Figure 4. Image processing pipeline for fingertip extraction.

We then find the local maxima by moving a 6.5 x 4.6 mm
window by 3.3 mm (horizontally) and 2.3 mm (vertically). If
there are multiple local maxima within 1.5 mm, we combine
them into a single maxima at the center point. In a second
step, we apply a 5 mm radius circular mask around each local
maxima; if they meet, we pick the highest maxima as thefingertip. If the local maxima falls at a screen edge, we
consider it as part of the grip and do not treat it as a fingertip.

Efficiency of Interaction vs. Comfort, Occlusion, Adaptability

Note that comfort and convenience trump efficiency in thisinteraction scenario; indeed, since our ad-lib controls must
respond to an approaching finger, they are unlikely to be as
fast as fixed controls that always remain on-screen. Yet
dedicated controls consume screen real estate, occlude thecontent, and cannot readily adapt to changing context (suchas one-handed interaction). Therefore our ad-lib controls
intentionally sacrifice some measure of efficiency in favor of
meeting these other demands of mobile interaction.

Thumb / Finger Distinction

We next calculate the orientation of the fingertip and use thisto identify whether it is a thumb or finger. To determine tilt,
we fit rotated bounding boxes (Figure 4e, yellow box) to the4surrounded by other ancillary controls, including a verticalslider for volume control. Having a full set of controls comeup in this case makes sense, because an index finger poised
above the screen is nimble enough to reach a variety oflocations. Furthermore, the two-handed usage posture
indicates the user is engaged with the system—and likely has
more cognitive and motor resources available—as opposed
to one-handed interaction scenarios.

Fade In Behavior: Respond Promptly

Of course, the video itself is the center of the experience. So
when the user is not interacting, there is no visible
interface—just the content. This is the default experiencethat we optimize for. But when users need to interact, wedon’t want them to feel like they have to wait around for thecontrols to appear. The response has to be snappy. As soon
as the system detects a hand approaching, it responds in a
speculative manner so that it can start presenting anappropriate interface promptly. Popping the controls into
existence would feel jarring, so we instead use a 200 ms fadein animation designed to draw the user’s eye to the core
playback controls: play/pause, rewind, and fast-forward.

One-Handed Interaction with the Thumb

We started our design with the idea that the ad-lib interface
fades in the UI when a finger approaches, and fades out the
UI when the finger moves away.
But our key insight was the following: since the interface
fades in and fades out anyway, it might as well fade in acontext-appropriate variation each time, which suits thecurrent grip, when the system senses the hand approaching.
Thus, when the user grips the device in a single hand andreaches over the screen with their thumb, the ad-lib interface
fades in a UI specifically designed for one-handed use. Since
it is hard to reach the center of the screen with the thumb, wefade in the controls closer to the edge, with a fan-shaped
layout that suits the natural movement of the thumb, and werender a version for either the right hand (Fig. 5b) or the left
(Fig. 5d), respectively. Furthermore, because one-handed
interaction is less dexterous and more suited to casual
activity, we fully render only a subset of the defaultinterface—the core playback controls.
We also provide dialing controls for the thumb (Fig. 6) thatallow the user to scrub through the timeline, or adjust thevolume. This illustrates how we take a graphical control(linear sliders) and translate them to a gestural interpretation
for the one-handed variant of the interface.

Figure 5. Ad-Lib Interface Controls, with (a) full controls
when the user approaches with an index finger, (b) a reduced
set of close-at-hand controls for one-handed interaction, (c)volume (vertical slider) flipped to the left when the user
approaches from the opposite side; (d) one-handed controls
fading in for the left hand; (e) a richer set of options fade inwith two-thumb operation; and (f) the controls fade out when
two fingers approach for pinch-to-zoom.
Fade Out Behavior: Withdraw Gracefully

When the finger moves out of range, the video player’s
controls fade, leaving the focus once again on the content.
However, for this fade-out transition we want the user’s
attention to drift back to the video, so our objective is towithdraw gracefully—like a good waiter slipping away when
his services are no longer needed. The system therefore
reacts deliberately, fading out the UI over a 1.2s animation.
Note that we also experimented with fading based directlyon the sensed finger proximity, similar to Medusa [1], but
this seemed to make the fade in / fade out feel less predictable
and more distracting visually than our fixed-time animations.

Figure 6. Dialing. The timeline and volume controls morph
into dials when they fade in for one-handed interaction.

Bimanual Grip with Index Finger: Controls at the Center

Note that our design makes no attempt to predict precisely
where the thumb will land. The controls always animate to
the same, fixed location that is a comfortable distance fromthe edge of the screen. We chose to do this for three reasons.
First, the difficulty of accurately predicting the landing
position from the early portion of a movement trajectory is

When the user grips the phone in one hand and approaches
the central areas of the screen with the index finger of the
opposite hand, we fade in the default full set of controls. The
fade-in animation, in this case, emphasizes the core playback
controls (they fade in over just 100 ms and expand as they doso, drawing the eye). The core playback controls are then5Calm Web Browser: Revelation of UI Affordances

well known. Second, we didn’t feel that further fine-tuning
the placement was necessary for typical small-screen mobile
scenarios; presenting the controls centered, near the edge, isgood enough on a small screen. Third, this makes the final
position of the controls completely predictable once the user
is familiar with them. An experienced user can therefore aim
for a particular screen location out of habit, without fully
attending to the graphical feedback.

Web pages employ various visual conventions to provide
affordances for actionable content. Links are underlined,hashtags are highlighted, and playback controls are overlaid
on interactive media such as videos or podcasts. But showing
all of these affordances can add a lot of clutter to the content
itself, whereas pages that omit such bells and whistles indeference to a cleaner design can leave the user uncertain of
which content is interactive.

As one final design flourish, when the one-handed controls
animate onto the screen, they follow a path that mimics thefinger approach. This helps to reinforce the connection
between the one-handed version of the controls and thecoming and going of the thumb from the screen.

On desktop web browsers, mouse-over often lights up
items—as can hover for touch as well [16,47,51])—but if the
input is treated as a single point, the user must resort totedious serial interrogation to figure out what can be tapped.
We implemented a mock-up of a web browser to explore useof the pre-touch modality to provide a more ‘calm’ webbrowsing experience—one that is free of such clutter in thereading part of the experience, allowing the user to enjoy aclean web page while holding (and reading from) the device.

Two-Thumb Interaction: Advanced Controls for 2nd Thumb

When the user reaches onto the screen with a second thumb,the ad-lib interface supplements the one-handed controls
with an additional set of advanced options (Fig. 5e). These
only slide in for the second thumb. The first thumb always
invokes the one-handed version of the UI described above.
Pinch-to-Zoom Variation

Of course, two-thumb interaction is just one way of using
two fingers on a touchscreen; if we sense the fingers
approaching in a pinch-to-zoom zoom posture, we fade out
the interface and present a gestural guide (Fig. 5f) instead.
While pinch-to-zoom is familiar to most users these days,
this approach could be used to reveal additional multi-touchgestures—an example of which we present in the next
section, on our “calm” web browser.
Approach Direction

We use the approach direction in several ways. For example,
as mentioned above, the one-handed variant of the ad-lib
interface slides into the screen in a path that mimics theapproach of the thumb. The approach trajectory also refines
the presentation of the vertical volume slider for thebimanual grip with the index finger (with the controls at thecenter of the screen). If the index finger approaches from theright, the volume slider appears to the right of the main
controls (Fig. 5a). But if it approaches from the left,
indicative of left-handed use, the volume slider flips to the
opposite side to make it easier to reach (Fig. 5c).

Figure 7. Our calm web browser reveals interactive affordances
in a nuanced way that feathers off with the finger contours.

When the user’s finger(s) approach the screen, the hyperlinks
and playback controls reveal themselves—and in a rich waythat feathers off with the contours of the finger, thumb, oreven the whole hand waving above the screen.
This feathering (gradual trailing-off) of the interactive
affordances allows the user to quickly see many actionable
items, rather than visiting them one-by-one. Furthermore,this emphasizes the items nearby, while more distant items
are hinted at in a subtle manner (Fig. 7). This leads to gradualrevelation of the affordances, in accordance with proximity
to the hand, rather than having individual elements visually“pop” in and out in a way that would be distracting; forexample, note how the video playback control (at the upper
right of Fig. 7) blends in a subtle way onto the page, rather
than popping in as a discrete object.

Summary of the Ad-Lib Interface

All of these nuances illustrate the many ways that the ad-lib
interface combines various aspects of grip, the number offingers, and the approach trajectory to optimize how the UI
presents itself. Multiple variations of the interface come andgo depending on the context, and carefully craftedanimations make the interface responsive (on approach) yet
unobtrusive (on fade-out) as appropriate. These accommodations are directed at comfort and convenience in mobileinteraction, particularly taking into account one handed
interaction for example, rather than efficiency per se, thereby
resulting in a novel user experience that uses the backgroundsensing capabilities afforded by pre-touch to tailor theinteraction to various contexts of mobile interaction.

We implement this effect by alpha-blending an overlay
image, containing the various visual affordances, with thethresholded and interpolated raw finger image (Figure 4c).
The overlay appears immediately when a finger comes into
proximity, and transitions from fully transparent to fully
visible as the hand moves closer to the screen.

6

it suggests that the fine-adjustment phase may be limited orabsent when acquiring large targets. In the following
sections, a pair of techniques illustrate how we mightleverage this distinction to enrich mobile interaction.

Freitag et al. [16] demonstrate hand shadows, but we givethis a fresh twist by using the hover profile to selectively
reveal interactive affordances, in a way that is truly multitouch and corresponds to the sensed posture of the fingers.

single-finger distance above touchscreen
30mm

Self-Revelation of Multi-Touch via Gesture Guides

Our web browser mock-up supports a two-finger tabbing
gesture to slide back and forth between browsing tabs. To
afford self-revelation of this gesture, the system fades in a
gesture overlay when it senses two fingers side-by-side in theappropriate posture for 100 ms (Fig. 8a). At the same time
the hyperlinks (and other visual affordances) fade out. Notethat Medusa [1] also reveals a fixed gesture guide when thearm hovers at the tabletop periphery, whereas ours appears
in-context and is contingent on the posture of the fingers.

temporal trajectory
for a large target
temporal trajectory
for a small target-1s-800ms20mm
10mm-600ms-400ms-200ms0ms(touch down)

Figure 9. Example pre-touch trajectories from one pilot user
for a small target (5x5 mm) versus a large target (40x40 mm).
The small target requires fine adjustment, whereas the finger
can “dive” to the large target with a purely ballistic motion.
Ballistic vs. Fine Tap: a Twitter Application Mockup

We implemented a mock-up of a mobile Twitter application
to illustrate this idea in practice. Like many mobile apps, thisuse case provides a long list of large targets (the tweets
themselves) that are mixed in with much smaller controls(the reply, retweet, and favorite icons).

Figure 8. The multi-finger gesture guide (a) and highlights forcollaboration using the sensed finger contour (b).
Finger Contour Highlighting for Collaborative Reference

Two problems present themselves. First, when the user taps
on a large target (a tweet, to see its full contents), thisimprecise, ballistic action may just happen to land on one of
the small icons, triggering an accidental and unwanted action(Fig. 10a). Second, when the user attempts to tap on the verysmall icons, if the user misses even by a few pixels (which iseasy to do with a fat finger, as shown in Fig. 10b) this instead
expands the tweet, which was not the intended operation. In
a sense the problem arises because the small targets nest
within the visual gestalt of the large one, the tweet itself.

We also support a collaboration mode where the sensed handcontour can be used to highlight portions of the page (Fig.
8b). The ability to easily refer to areas of a workspace (forexample using hand shadows) has previously been shown tobe vital to collaboration [52,53]; our highlighting featuredemonstrates how pre-touch could be used to realize this formobile devices. The yellow highlight is more expressive thana simple spotlight: it conforms to the contours of the fingers.
RETROACTIVE USES OF PRE-TOUCH

Pre-touch can also act as a back-channel that augments touch
events, by retroactively inspecting the approach trajectory at
the time of the touch-down event to glean more information
about the pointing movement. As such, this way of using pretouch resides in the background: it supports the foregroundaction (the intentional act of touching) in a way that is
invisible to the user. Said another way, unlike theanticipatory techniques described in the preceding section,
retroactive techniques produce no effect if the user doesn’tcomplete the movement and make contact with the screen.

To distinguish these cases, we inspect the in-air approachtrajectory upon the finger-down event. If we observe that thefinger motion was purely ballistic, we dispatch the tap eventto the large target (Fig. 10a). If the motion appears to include
fine adjustments, we instead dispatch it to a small target if
one lies within 7.7 mm of the finger-down event (Fig. 10b).

Our insight is that the approach trajectory provides additional
information that may help to better reveal the user’s intent.
The example aiming movements shown in Fig. 9, whichwere recorded for a right-handed pilot user tapping on targets
with his index finger while holding the phone in the
nonpreferred hand, provide an illustration of this. Whentapping on a small target, the user makes fine adjustments
prior to tapping down. But for a large target, the finger
simply lands on the screen with a ballistic motion.

Figure 10. Twitter mockup. (a) Each tweet (large target)
contains reply, retweet, and favorite icons (small targets).
(b) Precise pointing redirects to a nearby small target.

At present, we identify the fine-adjust phase by looking for a
touch trajectory with an altitude under 10 mm above thescreen, and within 15 mm of the touch-down location, for the
250 ms before the finger makes contact. This is a global
setting that was chosen heuristically for the bimanual grip,with the index finger used to acquire the target. As ourforthcoming informal evaluation reveals, this heuristic

Although as of this writing the effectiveness of such
retroactive interpretations lacks formal empirical support,
our observation is in accordance with the two-phase modelof pointing [40,41,61]. And like probabilistic pointing [19],7probably could be improved by optimizing it for one-handed
grips as well as on a per-user basis; different users appear tohave varying confidence (or tolerance for errors) when theyacquire small targets. Nonetheless, even in its present form
this technique provides an intriguing example of applying a
retroactive interpretation to pre-touch.
Flick vs. Select Discrimination

We explored a second example that uses this same insight todistinguish between flick (scrolling) and select (for a passage
of text) at the moment the finger comes into contact with thescreen. This dispenses with the need to separate thesetransactions by a tedious tap-and-hold interaction, which isstandard practice in touchscreen interfaces. We interpret anapproach trajectory with a ballistic swiping motion as a flick.
But selecting a passage of text requires a fine acquisition
phase to target the correct word boundary. We therefore canimmediately trigger text selection for such movements.

Figure 11. Hybrid touch+hover. (a) Selecting an icon with the
thumb while moving a second finger into range calls up aconvenient context menu. (b) With a one-handed grip, themenu gracefully degrades to a thumb-activated variant.

Hence, the technique offers three potential benefits: (1) itshortcuts the time-out that would otherwise be required by
tap-and-hold; (2) it calls up the menu at a convenient, userspecified location directly under the finger; and (3) it phrases
together selection and action (calling up the menu) into asingle compound task [6].

HYBRID TOUCH + HOVER INTERACTIONS

Finally, pre-touch lends itself to hybrid touch + hovergestures, which combine on-screen touch with simultaneousin-air gesture. This brings to light a little-explored class ofgesture—but previous work has used nonpreferred-handtouch to “nail down” [29] tabletop modes while the preferredhand makes in-air movements to manipulate 3D parameters.

While the technique is predominately designed to select theicon with the thumb, and then pick the command with theindex finger, to accommodate icons near the top of the screen
the user can alternatively touch down with the index finger
first (to select) and then pick from the menu with the thumb.
Our implementation supports either way of articulating the
gesture.

These hybrid gestures clearly reside in the foreground, yet
the example below illustrates how bringing in the
background sensing perspective affords graceful degradation
to a one-handed version of the technique.

Also, to clarify why hover is necessary (as opposed to “Pinand-Cross” [36] style interactions on a second touch), ourHybrid Menu uses foreknowledge of the second,
approaching finger to reveal the menu options before the user
has to commit to anything—and to do so in the right place,and without the finger fully occluding the screen location.

Hybrid Menu Combining Touch and Hover (and Grip)

We implemented a mock-up of a mobile file explorer, with agrid of icons (files) that support commands such as Copy,
Delete, Rename, and Share. Of course, these commands aremeaningless unless a file is selected first. Hence this is acompound task: users must first select the file, and only thencan they pick the command that acts on that object.

One-Handed Variation for Picking Commands with a Thumb

The menu activation gesture described above only makes
sense when using a mobile with both hands.
But because pre-touch affords sensing grip, the systemknows when the user is interacting one-handed. Thus, in thissituation, the technique gracefully degrades to enable menu
activation with a single thumb.

Traditionally, on mobile devices the user performs the selectsubtask with a tap-and-hold gesture on the desired object.
Tap-and-hold with a typical 1000 ms time-out is a widelyused but slow way to switch modes [34], yet the standard
vocabulary of mobile interaction offers few alternatives.

In this case, we have not devised any clever means to shortcircuit the timeout, so the user must tap-and-hold on thedesired icon with the thumb. This then activates the menu.
The system knows the thumb was used in this case, so it
presents the menu with a fan-shaped layout that arcs in a
direction appropriate to the side (right or left) that the thumbapproaches from. The user then picks the desired command.

We therefore implemented a hybrid touch+hover gesture(Fig. 11) that integrates selection of the desired object withthe activation of the menu—articulated as a single compoundtask. The user first selects the desired file by holding a thumbon it, while simultaneously bringing a second finger into
range. This immediately summons the object’s menu.
Furthermore, since the system knows where the user’s finger
is, it can invoke the menu at a convenient location, directly
under the finger. The opacity of the menu is proportional to
the finger’s altitude above the display. The user thencompletes the transaction by touching down on the desiredcommand. Alternatively, the user can cancel the action
simply by lifting the finger.

Possibilities for Mobile Gaming

We implemented a simple prototype of a soccer game toillustrate the potential of hybrid touch+hover for gaming(Fig. 12). The game uses the fingers to mime kicking a soccer
ball: one finger stays planted, while the other finger strikes
the ball. The 3D trajectory of the kick depends on the
direction of movement and how high (or low) to the groundthe user kicks. The finger can also be lifted above the ball to8Ad-lib Interface (Video Player). Users appreciated that thecontrols got out of the way (didn’t block the video) while
viewing content; as one user commented, “I like thetransparent controls, and they’re predictive.” Users
particularly liked the facile transition to one-handed
interaction, which “feels very natural to my hand” and allows
using “a single hand in a comfortable position.” Users also
really liked the transformation of the volume and timeline
controls to a dialing gesture. A couple of users expressed adesire for the controls to respond (appear or transition) more
quickly to their grips. One user with large hands felt that the
one-handed controls were too close to the edge. In regards totwo-thumb interaction, another user felt that the core
playback controls should always appear for the right thumb,with the advanced controls always on the left thumb, rather
than bringing up the playback controls for whichever thumbapproaches first. When trying pinch-to-zoom, one usersuggested another dialing control for one-handed zooming.

“step” on it, or to move over it in order to back-foot the ball,for example. The phone vibrates when the finger hits the ball.
Other possible uses include controlling an avatar, or sensing
walking-in-place interactions for virtual navigation [31].

Figure 12. Soccer game. (a) Striking the ball along a 3D
trajectory. (b) Moving over the ball does not strike it.
INFORMAL EVALUATION

To gain some initial insight into our interaction techniques,
we had test users try them out and offer some preliminary
feedback. Participants tried all applications described above
except for the soccer game.

Calm Web Browser. Users liked the clean design for reading
and found it “really helpful to see hyperlinks in an efficientway” so that “I know exactly what I need when browsing the
web page.” Users also appreciated clear information on thecontent type (video, images, links). Several users commented
that the graphic design of our revealed hyperlinks could be
subtler and “more transparent with less emphasized borders.”
Thus, the reading experience satisfies our ‘calm’ design goal,
but the hyperlink overlays are perhaps more distracting than
we intended—although it would be straightforward to tonethat down slightly. Users appreciated the guide for the tab
switch gesture, but also felt that it should eventually be
suppressed because it is only be useful for first-time users.

Participants

We recruited 7 participants (3 female, 4 male) aged between
23-31 (average 27) years. All used a touchscreen mobile
phone every day, and had owned one for more than 2 years.
Procedure

Participants used the phone while seated, resting their arms
on a table, and were allowed to use the phone as they foundcomfortable (e.g. one-handed with a thumb vs. two handed,using an index finger to point, or interleaving the two as
desired). However, we did also ask users to try the interfaces
using the various grips supported. We also interviewed users
regarding each technique. The study took about an hour;participants were compensated with a $10 cafeteria coupon.

Ballistic vs. Fine Tap and Flick vs. Select. Users tried thesetechniques both bimanually (holding the device in the
nonpreferred hand while using the index finger of thepreferred hand to point) as well as one-handed. Reactions
were divided. For some users, the interactions seemed to bewell-tuned to how they naturally pointed at small targets,making it “an elegant solution to handling low-resolution
thumbs” and a technique that “helps me avoid tapping on thewrong things.” Other users “would need some time to adjust
to it” or felt “it didn’t work well with my fingers.” This hints
that these interventions can succeed with appropriate design,
but per-user and/or per-grip settings (rather than the global
time and motion thresholds that we currently employ) may
be necessary to accommodate users’ varied styles of pointing
at small targets. Clearly, empirical studies will be necessary
to sort out these issues and unpack the technique’s potential.

For the video player (with ad-lib interface controls), webbrowser (with calm revelation of hyperlinks), and file
explorer (with the hybrid touch+hover menus), we briefly
explained and demonstrated each interaction technique.
Participants then tried the techniques on their own. But for
the Ballistic vs. Fine Tap and Flick vs. Select discrimination,
since the intervention is supposed to be completely invisible,
we simply asked users to tap on various targets (for theTwitter mock-up) or scroll and select passages of text (forflick vs. select) without any prior explanation of thetechniques. After users tried them for a while, only then did
we disclose how they worked. Users then had a final
opportunity to experiment with them further.

Hybrid touch+hover gesture for menus. Users liked that this“pinch context menu” helps to “shorten the selection time,”
allowing them to “go to the buttons faster and morenaturally.” Users also really liked that the techniqueautomatically senses grip to accommodate one-handed
interaction: “adapting to the hand position is great.”
However, we also observed that our gesture recognition forthis action has some quirks that caused false-positive

Results

All participants were able to learn the techniques within a
few attempts—even the touch+hover hybrid gesture to callup a menu, which at present clearly requires an initialdemonstration for users to discover the technique. Overall,participants responded positively to the techniques.

9between different, context-appropriate presentations thattake into account both grip and hover).

appearances of the menu for some users. Users could easily
escape the menu by withdrawing their hand, but this added
effort was annoying when it occurred.

But this is also apparent in the grip-contingent aspects of ourHybrid (touch+hover) menu, and in the way our findings
suggest a natural extension of the Retroactive interpretationof Ballistic vs. Fine Tap to take into account grip as well.
Nonetheless the Calm Web Browser, Soccer Game, and ourpresent implementation of Ballistic vs. Fine Tap (and its
Flick vs. Select variation) use only hover, but in new andinteresting ways, to extend the themes of our research.

Overall reactions. No technique stood out as a universal
favorite, yet almost all of the techniques had strong
supporters. The ad-lib interface (and particularly the onehanded version thereof), calm web browser, ballistic vs. finetap, and automatic presentation of one-handed context menus
in the file explorer were all explicitly mentioned as favorites.
However, the Ballistic vs. Fine Tap (and Flick vs. Select)exhibited a clearly bimodal response, as some users foundthe techniques completely natural while others found them
ill-suited to their typical way of selecting small targets. And
while users were able to learn the “pinch context menu” that
we explored fairly quickly, the need to learn a new,unfamiliar gesture for this caused a majority of participants
to rank this technique slightly lower than the others.

An interesting future direction would be to employ pre-touch
hardware to explore unencumbered aiming movements onmobile devices in detail. Our exploration of Ballistic vs.
Finely Targeted taps hints at some insights that might berevealed by such a study, but a much deeper analysis thatlooks at a variety of mobile contexts (and one-handed
interaction in particular) is called for. As user comments
revealed, our distinction between ballistic and finelytargeted taps likely requires a grip-contingent model (among
other possible refinements) to meet wider success.

CONCLUSION AND FUTURE WORKThe sensor that we employed for our explorations has whatmight be viewed as a quirk: the touchscreen senses both grip
and hover. But as our explorations have demonstrated, thisapparent “quirk” actually presages a deeper insight. To use aphone during everyday mobile activity, the first thing one
must do is pick it up, and hold it, with a particular grip on thedevice—which of course involves contact of the hand andfingers. Grip therefore precedes interaction with the screen
itself through ‘touch.’ Likewise, traditional touch events fireat the moment one makes contact with the digitizer, yet thegenesis of the grasping or aiming movement comes muchearlier, and originates away from the screen itself.

Exploring pre-touch on other form-factors could also yield
new techniques. For example, one direction would be pretouch sensing for tablets, where the larger screen brings
about a greater diversity of grips, which therefore might
demand different approaches to some of the design decisions
than we made for our handheld form-factor. In particular,
better prediction of the touch-down location from the grip
and approach trajectory might be necessary to effectively
support anticipatory techniques on a larger screen.
Clearly, the unification of grip and hover as pre-touch raises
many possibilities for direct-touch interaction. While wehave concerned ourselves particularly with the opportunities
this emerging sensing modality opens up for a few common
problems that users encounter when using mobile devices,pre-touch appears to offer much promise in addressing
additional issues in mobile interaction as well. Future
explorations of pre-touch can explore, study, and analyze
these and many other possibilities—both expected andwholly unanticipated—that surely await discovery if one
only looks not only under, but also around and above theright stones.

Therefore, this sensor and its quirk—the seemingly
incidental unification of grip and hover afforded by selfcapacitance—compelled us to conceive of ‘touch’ in a waythat embraces these natural human behaviors and thatfurthermore fully leverages them to add more contextual
richness to mobile touch interaction. We signified this shift
in perspective by envisioning this emerging modality as pretouch, a term that properly frames this channel as an umbrella
for both grip and hover, and that fosters its conception as asensing modality that augments and enhances normal touch
inputs from the background of the interaction.
The thread connecting our contributions has been theobservation that multi-touch hover and grip, as afforded by
self-capacitive touchscreens, raises many possibilities—andparticularly in a mobile setting, with an emphasis oncontextual sensing in the background. We conceptually unify
grip and hover under pre-touch, a perspective whichsignificantly extends the most closely related works (e.g.
[1,9,16,60]) that leverage the pre-input stages articulated byFreitag et al. [16]. Some techniques we chose to explore were
motivated by common problems in mobile interaction, wherere-framing these as problems of missing context led to novel
techniques. Additionally, several techniques combine
grip+hover, most notably the Ad-Lib Interface (which goesbeyond previous work by morphing the entire mobile UIRIGHTS FOR FIGURES
Figures 1 and 3-12 © Ken Hinckley, 2016.

REFERENCES

1. Michelle Annett, Tovi Grossman, Daniel Wigdor,
George Fitzmaurice. 2011. Medusa: A Proximity-Aware
Multi-touch Tabletop. In Proceedings of the 24th
annual ACM symposium on User interface software andtechnology (UIST '11), 337-346.
http://dx.doi.org/10.1145/2047196.2047240
2. Michelle Annett, Anoop Gupta, Walter F. Bischof.
2014. Exploring and Understanding Unintended Touchduring Direct Pen Interaction. ACM Trans. Comput.-10and surfaces (ITS '12), 355-358.
http://doi.acm.org/10.1145/2396636.2396699.
13. S. H. Creem and D. R. Proffitt. 2001. Grasping objects
by their handles: A necessary interaction between
cognition and action. Journal of Experimental
Psychology: Human Perception and Performance 27:
218-228.
14. Paul H. Dietz, Benjamin Eidelson, Jonathan Westhues,
Steven Bathiche. 2009. A practical pressure sensitive
computer keyboard. In Proceedings of the 22nd annual
ACM symposium on User interface software andtechnology (UIST '09), 55-58.
http://doi.acm.org/10.1145/1622176.1622187.
15. Fogale Nanotech. Fogale Sensation Technology.
Retrieved September 22, 2015 from: http://www.fogalesensation.com/technology.
16. Georg Freitag, Michael Tränkner, Markus Wacker.
2012. Enhanced feed-forward for a user aware multitouch device. In Proceedings of the 7th Nordic
Conference on Human-Computer Interaction: Making
Sense Through Design (NordiCHI '12), 578-586.
http://doi.acm.org/10.1145/2399016.2399104.
17. M. Goel, A. Jansen, T. Mandel, S. Patel, . N., J. O.
Wobbrock. 2013. ContextType: using hand posture
information to improve mobile touch screen text entry.
In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '13), 2795-2798.
http://doi.acm.org/10.1145/2470654.2481386.
18. Mayank Goel, Jacob Wobbrock, Shwetak Patel. 2012.
GripSense: Using Built-In Sensors to Detect Hand
Posture and Pressure on Commodity Mobile Phones. In
Proceedings of the 25th annual ACM symposium on
User interface software and technology (UIST '12), 545554. http://doi.acm.org/10.1145/2380116.2380184.
19. Tovi Grossman and Ravin Balakrishnan. 2005. A
probabilistic approach to modeling two-dimensional
pointing. ACM Trans. Comput.-Hum. Interact. 12, 3(September 2005): 435-459.
http://doi.acm.org/10.1145/1096737.1096741.
20. Tovi Grossman, Ken Hinckley, Patrick Baudisch,
Maneesh Agrawala, Ravin Balakrishnan. 2006. Hover
widgets: using the tracking state to extend thecapabilities of pen-operated devices. In Proceedings of
the SIGCHI Conference on Human Factors in
Computing Systems (CHI'06), 861-870.
http://doi.acm.org/10.1145/1124772.1124898.
21. Yves Guiard. 1987. Asymmetric division of labor inhuman skilled bimanual action: The kinematic chain asa model. Journal of Motor Behavior 19, 4: 486-517.
22. Beverly L. Harrison, Kenneth P. Fishkin, Anuj Gujar,
Carlos Mochon, Roy Want. 1998. Squeeze me, hold me,tilt me! An exploration of manipulative user interfaces.
In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI'98), 17-24.
http://doi.acm.org/10.1145/274644.274647.

Hum. Interact. 21, 5: Article 28 (39pp).
http://doi.acm.org/10.1145/2674915
3. Victoria Bellotti, Maribeth Back, W. Keith Edwards,
Rebecca E. Grinter, Austin Henderson, Cristina Lopes.
2002. Making sense of sensing systems: five questions
for designers and researchers. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems (CHI '02), 415-422.
http://doi.acm.org/10.1145/503376.503450.
4. Joanna Bergstrom-Lehtovirta and Antti Oulasvirta.
2014. Modeling the functional area of the thumb on
mobile touchscreen surfaces. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems (CHI '14), 1991-2000.
http://doi.acm.org/10.1145/2556288.2557354.
5. Peter Brandl, Jakob Leitner, Thomas Seifried, Michael
Haller, Bernard Doray, Paul To. 2009. Occlusion-aware
menu design for digital tabletops. In CHI '09 Extended
Abstracts on Human Factors in Computing Systems(CHI EA '09), 3223-28.
http://doi.acm.org/10.1145/1520340.1520461.
6. W. Buxton. 1986. Chunking and Phrasing and the
Design of Human-Computer Dialogues. In Proceedingsof the IFIP World Computer Congress, 475-480.
7. W. Buxton. 1995. Integrating the Periphery andContext: A New Taxonomy of Telematics. In
Proceedings of Graphics Interface '95, 239-246.
8. William Buxton. 1990. A three-state model of graphicalinput. In Proceedings of the IFIP TC13 Third
Interational Conference on Human-Computer
Interaction, 449-456.
9. Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison,
Jennifer Mankoff, Scott E. Hudson. 2014. Air+touch:interweaving touch & in-air gestures. In Proceedings ofthe 27th annual ACM symposium on User interface
software and technology (UIST '14), 519-525.
http://doi.acm.org/10.1145/2642918.2647392.
10. Lung-Pan Cheng, Meng Han Lee, Che-Yang Wu, FangI Hsiao, Yen-Ting Liu, Hsiang-Sheng Liang, Yi-Ching
Chiu, Ming-Sui Lee, Mike Y. Chen. 2013.
iRotateGrasp: automatic screen rotation based on graspof mobile devices. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems(CHI '13), 3051-3054.
http://doi.acm.org/10.1145/2470654.2481424
11. Lung-Pan Cheng, Hsiang-Sheng Liang, Che-Yang Wu,
Mike Y. Chen. 2013. iGrasp: grasp-based adaptive
keyboard for mobile devices. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems (CHI '13), 3037-3046.
http://doi.acm.org/10.1145/2470654.2481422.
12. Victor Cheung, Jens Heydekorn, Stacey Scott, Raimund
Dachselt. 2012. Revisiting hovering: interaction guides
for interactive surfaces. In Proceedings of the 2012
ACM international conference on Interactive tabletops

11Youngbeom Lee, Sangryong Kim. 2006. Hand Grip
Pattern Recognition for Mobile User Interfaces. In
Proceedings of the 18th conference on Innovative
applications of artificial intelligence - Volume 2(IAAI'06), 1789-1794.
33. Sven Kratz and Michael Rohs. 2009. HoverFlow:
expanding the design space of around-device
interaction. In Proceedings of the 11th International
Conference on Human-Computer Interaction with
Mobile Devices and Services (MobileHCI '09), Article 4, 8 pp. http://doi.acm.org/10.1145/1613858.1613864.
34. Yang Li, Ken Hinckley, Zhiwei Guan, James A.
Landay. 2005. Experimental analysis of mode switching
techniques in pen-based user interfaces. In Proceedingsof the SIGCHI Conference on Human Factors in
Computing Systems (CHI '05), 461-470.
http://doi.acm.org/10.1145/1054972.1055036.
35. Shenwei Liu and François Guimbretière. 2012.
FlexAura: A Flexible Near-Surface Range Sensor. In
Proceedings of the 25th annual ACM symposium on
User interface software and technology (UIST '12), 327330. http://doi.acm.org/10.1145/2380116.2380158
36. Yuexing Luo and Daniel Vogel. 2015. Pin-and-Cross: A
Unimanual Multitouch Technique Combining Static
Touches with Crossing Selection. In Proceedings of the
28th Annual ACM Symposium on User Interface
Software & Technology (UIST '15), 323-332.
http://dx.doi.org/10.1145/2807442.2807444.
37. Christine Mackenzie and Thea Iberall. 1994. The
Grasping Hand. Advances in Psychology 104, ed. G.
Stelmach and P. Vroon. North Holland.
38. Nicolai Marquardt, Ricardo Jota, Saul Greenberg,
Joaquim A. Jorge. 2011. The Continuous Interaction
Space: Interaction Techniques Unifying Touch andGesture on and Above an Interaction Surface. In
Proceedings of the 13th IFIP TC 13 international
conference on Human-computer interaction - Volume
Part III (INTERACT'11), 461-476.
39. R. G. Marteniuk, C. L. MacKenzie, M. Jeannerod, S.
Athenes, C. Dugas. 1987. Constraints on human arm
movement trajectories. Canadian Journal of Psychology
41, 3: 365-378.
40. Michael J. McGuffin and Ravin Balakrishnan. 2005.
Fitts' law and expanding targets: Experimental studies
and designs for user interfaces. ACM Trans. Comput.Hum. Interact. 12, 4 (December 2005): 388-422.
http://doi.acm.org/10.1145/1121112.1121115.
41. David E Meyer, Richard A Abrams, Sylvan Kornblum,
Charles E Wright, J. E. Keith Smith. 1988. Optimality
in human motor performance: ideal control of rapid
aimed movements. Psychological Review 95: 340-370.
42. Matei Negulescu and Joanna McGrenere. 2015. Grip
Change as an Information Side Channel for Mobile
Touch Interaction. In Proceedings of the 33rd Annual
ACM Conference on Human Factors in Computing

23. Seongkook Heo and Geehyuk Lee. 2011. Forcegestures: augmenting touch screen gestures with normal
and tangential forces. In Proceedings of the 24th annual
ACM symposium on User interface software andtechnology (UIST '11), 621-626.
http://doi.acm.org/10.1145/2047196.2047278.
24. Christopher F. Herot and Guy Weinzapfel. 1978. OnePoint Touch Input of Vector Information from
Computer Displays. In Proceedings of the 5th annual
conference on Computer graphics and interactive
techniques (SIGGRAPH '78), 210-216.
http://doi.acm.org/10.1145/800248.807392.
25. Otmar Hilliges, Shahram Izadi, Andrew D. Wilson,
Steve Hodges, Armando Garcia-Mendoza, Andreas
Butz. 2009. Interactions in the air: adding further depth
to interactive tabletops. In Proceedings of the 22nd
annual ACM symposium on User interface software andtechnology, 139-148.
http://doi.acm.org/10.1145/1622176.1622203.
26. K. Hinckley, M. Pahud, H. Benko, P. Irani, F.
Guimbretiere, M. Gavriliu, X. Chen, F. Matulic, B.
Buxton, A. Wilson. 2014. Sensing Techniques for
Tablet+Stylus Interaction. In Proceedings of the 27th
annual ACM symposium on User interface software andtechnology (UIST'14), 605-614.
http://dx.doi.org/10.1145/2642918.2647379.
27. Ken Hinckley, Jeff Pierce, Eric Horvitz, Mike Sinclair.
2005. Foreground and Background Interaction with
Sensor-Enhanced Mobile Devices. ACM Trans.
Comput.-Hum. Interact. 12, 1 (Special Issue on SensorBased Interaction) (March 2005): 31-52.
http://doi.acm.org/10.1145/1057237.1057240.
28. Christian Holz and Patrick Baudisch. 2010. The
generalized perceived input point model and how todouble touch accuracy by extracting fingerprints. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '10), 581-590.
http://doi.acm.org/10.1145/1753326.1753413.
29. Bret Jackson, David Schroeder, Daniel F. Keefe. 2012.
Nailing down multi-touch: anchored above the surface
interaction for 3D modeling and navigation. In
Proceedings of Graphics Interface 2012 (GI '12), 181184.
30. A. Karlson, B. Bederson, J. Contreras-Vidal. 2006.
Understanding single-handed mobile device interaction,
in Handbook of research on user interface design andevaluation for mobile technology, 86-101.
31. Ji-Sun Kim, Denis Gračanin, Taeyoung Yang, Francis
Quek. 2015. Action-Transferred Navigation Technique
Design Approach Supporting Human Spatial Learning.
ACM Trans. Comput.-Hum. Interact 22, 6: Article 30(September 2015), 42 pages. .
http://dx.doi.org/10.1145/2811258.
32. Kee-Eung Kim, Wook Chang, Sung-Jung Cho,
Junghyun Shim, Hyunjeong Lee, Joonah Park,

12

the 2010 ACM conference on Computer supported
cooperative work (CSCW '10), 271-280.
http://doi.acm.org/10.1145/1718918.1718969.
53. John C. Tang and Scott Minneman. 1991.
VideoWhiteboard: video shadows to support remotecollaboration. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems(CHI '91), 315-322.
http://doi.acm.org/10.1145/108844.108932.
54. Brandon T. Taylor and Jr. V. Michael Bove. 2009.
Graspables: Grasp-Recognition as a User Interface. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '09), 917-926.
http://doi.acm.org/10.1145/1518701.1518842.
55. Daniel Vogel and Ravin Balakrishnan. 2010. Occlusionaware interfaces. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems(CHI '10), 263-272.
http://doi.acm.org/10.1145/1753326.1753365.
56. D. Wigdor and D. Wixon. 2011. Design Guidelines:
Self-Revealing Multi-Touch Gestures, in Brave NUI
world: designing natural user interfaces for touch andgesture. Elsevier, 150-154.
57. Raphael Wimmer and Sebastian Boring. 2009.
HandSense - Discriminating Different Ways of
Grasping and Holding a Tangible User Interface. In
Proceedings of the 3rd International Conference on
Tangible and Embedded Interaction (TEI '09), 359-362.
http://doi.acm.org/10.1145/1517664.1517736.
58. Katrin Wolf, Christian Müller-Tomfelde, Kelvin Cheng,
Ina Wechsung. 2012. PinchPad: performance of touchbased gestures while grasping devices. In Proceedingsof the Sixth International Conference on Tangible,
Embedded and Embodied Interaction (TEI '12), 103110. http://dx.doi.org/10.1145/2148131.2148155.
59. Katrin Wolf, Anja Naumann, Michael Rohs, Jörg
Müller. 2011. Taxonomy of Microinteractions: Defining
Microgestures based on Ergonomic and Scenariodependent Requirements. In Proceedings of the 13th
IFIP TC 13 international conference on Humancomputer interaction - Volume Part I (INTERACT'11),
559-575.
60. Haijun Xia, Ricardo Jota, Benjamin McCanny, Zhe Yu,
Clifton Forlines, Karan Singh, Daniel Wigdor. 2014.
Zero-latency tapping: using hover information to predict
touch locations and eliminate touchdown latency. In
Proceedings of the 27th annual ACM symposium on
User interface software and technology (UIST '14), 205214. http://doi.acm.org/10.1145/2642918.2647348.
61. Xing-Dong Yang, Tovi Grossman, Pourang Irani,George Fitzmaurice. 2011. TouchCuts and TouchZoom:
enhanced target selection for touch displays using finger
proximity sensing. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems

Systems (CHI '15), 1519-1522.
http://doi.acm.org/10.1145/2702123.2702185.
43. Mohammad Faizuddin Mohd Noor, Andrew Ramsay,
Stephen Hughes, Simon Rogers, John Williamson,
Roderick Murray-Smith. 2014. 28 frames later:
predicting screen touches from back-of-device gripchanges. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (CHI '14), 20052008. http://doi.acm.org/10.1145/2556288.2557148.
44. Halla B. Olafsdottir, Theophanis Tsandilas, Caroline
Appert. 2014. Prospective motor control on tabletops:
planning grasp for multitouch interaction. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '14), 2893-2902.
http://doi.acm.org/10.1145/2556288.2557029.
45. Henning Pohl and Roderick Murray-Smith. 2013.
Focused and casual interactions: allowing users to vary
their level of engagement. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems (CHI '13), 2223-2232.
http://doi.acm.org/10.1145/2470654.2481307.
46. Simon Rogers, John Williamson, Craig Stewart,
Roderick Murray-Smith. 2011. AnglePose: robust,
precise capacitive touch tracking via 3d orientation
estimation. In Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems (CHI '11),2575-2584.
http://doi.acm.org/10.1145/1978942.1979318.
47. Samsung. How Do I Use Air Gestures? Retrieved
September 23 from:http://www.samsung.com/us/support/howtoguide/N000
0003/10141/120552.
48. Itiro Siio and Hitomi Tsujita. 2006. Mobile interaction
using paperweight metaphor. In Proceedings of the 19th
annual ACM symposium on User interface software andtechnology (UIST '06), 111-114.
http://dx.doi.org/10.1145/1166253.1166271.
49. J. R. Smith, E. Garcia, R. Wistort, G. Krishnamoorthy.
2007. Electric field imaging pretouch for robotic
graspers. In IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS 2007), 676-683.
50. Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan
Fanello, Shahram Izadi, Cem Keskin, Otmar Hilliges.
2014. In-air gestures around unmodified mobile devices.
In Proceedings of the 27th annual ACM symposium on
User interface software and technology (UIST '14), 319329. http://doi.acm.org/10.1145/2642918.2647373.
51. Sony. Floating Touch--Developer World Mobile.
Retrieved from:http://developer.sonymobile.com/knowledgebase/technologies/floating-touch/.
52. Anthony Tang, Michel Pahud, Kori Inkpen, Hrvoje
Benko, John C. Tang, Bill Buxton. 2010. Three's
company: understanding communication channels inthree-way distributed collaboration. In Proceedings of

13

(CHI '11), 2585-2594.
http://doi.acm.org/10.1145/1978942.1979319.
62. Dongwook Yoon, Ken Hinckley, Hrvoje Benko,
François Guimbretière, Pourang Irani, Michel Pahud,
Marcel Gavriliu. 2015. Sensing Tablet Grasp + Micromobility for Active Reading. In Proceedings of the 28th
Annual ACM Symposium on User Interface Software &Technology (UIST '15), 477-487.
http://dx.doi.org/10.1145/2807442.2807510.
63. Chris Ziegler. Apple brings 3D Touch to the iPhone 6S.
Retrieved Sept. 9, 2015 from:http://www.theverge.com/2015/9/9/9280599/appleiphone-6s-3d-touch-display-screen-technology.

14