Understanding Mid-Air Hand Gestures: A Study of Human
Preferences in Usage of Gesture Types for HCI
Roland Aigner1, Daniel Wigdor2, Hrvoje Benko3, Michael Haller4, David Lindlbauer4, Alexandra Ion4,
Shengdong Zhao5, Jeffrey Tzu Kwan Valino Koh6
1

Ars Electronica Futurelab
Ars Electronica GmbH
Linz, Austria
roland.aigner@aec.at

2

4

5

Media Interaction Lab
University of Applied Sciences Upper
Austria, Hagenberg, Austria
mi-lab@fh-hagenberg.at

3

Department of Computer Science
University of Toronto
Toronto, ON
dwigdor@dgp.toronto.edu

Department of Computer Science
National University of Singapore
Singaporezhaosd@comp.nus.edu.sg

Microsoft Research
Microsoft Corporation
Redmond, WA
benko@microsoft.com
6

Interactive and Digital Media InstituteNational University of Singapore
Singapore
j.koh@nus.edu.sg

ABSTRACTIn this paper we present the results of a study of humanpreferences in using mid-air gestures for directing otherhumans. Rather than contributing a specific set of gestures,
we contribute a set of gesture types, which together make a
set of the core actions needed to complete any of our six
chosen tasks in the domain of human-to-human gesturalcommunication without the speech channel. We observed
12 participants, cooperating to accomplish different tasks
only using hand gestures to communicate. We analyzed
5,500 gestures in terms of hand usage and gesture type, using
a novel classification scheme which combines three existingtaxonomies in order to better capture this interaction space.
Our findings indicate that, depending on the meaning of thegesture, there is preference in the usage of gesture types, suchas pointing, pantomimic acting, direct manipulation,
semaphoric, or iconic gestures. These results can be used as
guidelines to design purely gesture driven interfaces forinteractive environments and surfaces.
INTRODUCTION

Recent advances in computer vision, particularly in realtime hand [26] and body tracking [23] are empoweringcomputers and intelligent environments to recognize humangestures from a distance. These novel technologies reducebarriers to interaction and increase the input bandwidth
between the user and the computer, without requiring theuser to wear or acquire a tracked object. Since surface andtabletop systems have expanded to include proximity
sensing [13], there is potential for computer vision based
mid-air interaction to become a part of tabletop computing[12] and large-scale displays [3]. The recognition of mid-air
gestures enables designers to create interfaces that enable
explicit control of such systems.
As is true of any new input modality, there is a need forextensive design of the physical primitives which make-up
the interaction [1]. Since the WIMP GUI is still a prevalent
metaphor in computer interfaces, a simple approach is to
Copyright Microsoft Corporation
Microsoft Research Technical Report MSR-TR-2012-111.

Figure 1. Two participants in our study: one was gesturing todirect the actions of the other. We excluded the audio channel
in order to force and observe common gestural behavior.

adopt its “point and click” and related primitives. For
example, Vogel and Balakrishnan [25] built a pointing-andclick interface for large screen displays which simulates
mouse input. They identified issues with precision in
pointing, ambiguity of finger movements, and lack ofphysical feedback. These issues are critical for the control
of a point-and-click, mouse-based UI, but offer little
guidance to designers wishing to develop new, possibly
more suitable input primitives. In particular suchtransformation may reduce much of the richness of the new
input modality, and sacrifice available bandwidth [8].
To create interfaces that leverage a gestural input paradigm,
there is a need to build gestural interfaces from scratch –including the design of new physical primitives. One
common approach has been to elicit gestures applying theuser defined interfaces (UDI) methodology. This technique
is useful in determining a novice’s “first guess”. However,researchers have found that there is little consensus amongusers in association between gestures and their expected
effect [24, 28]. In contrast, designed languages can beformed to avoid conflicts, but also to be ergonomic andhigh-bandwidth. In view of this, we believe that gesturelanguages need to be designed, rather than observed. A key

limitation, however, is the lack of guidelines to aid
designers in creating those languages.
To overcome this issue, we seek to bridge the gap between
those two methodologies by investigating what linguists
term gesture types [19]. Although gesticulation humans
perform during speech is quite different in its nature, webuild on this traditional linguistic practice for a more
systematic investigation of gestures in HCI. Rather thanseeking to construct a dictionary of particular gestures for agiven set of functions, classifying physical actions enables
us to create middle ground between elicited and designed
gestures. While a UDI methodology might provide viable
but inconsistent results, our approach enables for athorough design of a gesture language by providing
guidance that, for example, a gesture for a particular action
should be chosen to involve unimanual static hand posturesor bimanual movements, or that it should use pointing,descriptions of the objects, or semaphoric codes.
In order to produce our gesture types, we conducted astudy. Our methodology was similar to that of UDI, in thatparticipants were given a ‘before’ and ‘after’ situation,
where user-selected gestures were used to make thetransition between those two states [28]. Unlike previous
work, our study provided a recognizer – a second
participant, who was observing, via a muted video feed, theactions of the first participant. The first participant was
tasked with guiding the observer in completing various
tasks, using only gestures shown over a video feed (seeFigure 1), thus lacking speech. The ‘human’ recognizer
enabled participants in our study to perform gesturalinteraction within a closed feedback loop, enablingrefinement over time. The choice of using a human insteadof an artificial gesture recognizer was also taken to preventfrom biased behavior and to provide an unconstrained
environment to the directing participant. While this study
setup is similar to previous works in CSCW [10, 17], thegoals and information collected were different. In our
methodology the actor was meant to communicate only
using mid-air gestures, in order to accomplish a task in a‘virtual’ environment, just like in controlling an interactive
computing system.. Except for visual feedback showing theresult of the given input, the communication was
intentionally one-way, as with a unimodal gesture interface.
Additionally, the study was designed not to address highlevel human communication tasks, but low-level actions
such as manipulating objects, guiding avatars, anddescribing simple shapes, in order to collect gesture data foreach of them for further analysis. As we will discuss, therefinement process saw the introduction of new types ofgestures, ad hoc formalisms were created, and wholegesture languages were observed.
During this process, we collected approximately 5,500
gestures from 12 participants during 6 different scenarios.

We analyzed those in terms of expected effect and gesture
type, using our own classification scheme. We alsocollected data about the number of hands used, by splitting
into unimanual and bimanual gestures. The results indicatethat for each expected effect of a gesture, there are
preferences in both type and number of hands. These
preferences, reported in detail in this paper, represent twocontributions. First, they provide a baseline measurement of
conventional human behavior (after a short grounding
process) which goes beyond the ‘first guess’ provided by
UDI. Second, the results, in and of themselves, form the basis
for the development of guidelines which designers can use to
design their own gesture languages, taking into account not
only user preference, but all the factors relevant to thecontext of the language [27].
In this paper, we present the design of the study, ourclassification system, and our findings about gesture typeusage, including user choices for uni- vs. bimanual
gestures. Our contribution is our classification scheme, andknowledge of the suitability of gesture types for the chosen
gesture effects.
RELATED WORK
Gesture Classification

Many different classifications have been proposed forspeech-related hand movements [5, 19]. Many of these aremodifications of Efron’s [6] pioneering work about
conversational behavior between Jewish and Italianimmigrants in New York City. For example, Kendon [16]arranges gestures along a continuum, which reflects thegesture’s relation to accompanying speech. He proposedclasses of gesticulation (beat, cohesives), language-like(iconic), pantomimes, emblems (deictic), and sign language(symbolic), with the necessity of accompanying speech tofacilitate communication declining in this order.
Cadoz [5] classifies gestures according to their function into
semiotics, ergotic and epistemic gestures. McNeill [19]separates them into the classes of iconics, metaphorics, beats,cohesives, and deictics. These authors investigate themultidisciplinary research field of human gesturing. As aresult, their classifications are created for gestures
accompanied by speech, such as in narration, and are
therefore not appropriate for our domain. In contrast,
Karam and schraefel [15] created a more extensive
taxonomy, especially tailored for HCI, and serves as thebasis for our classification. For what they call ‘gesture
styles’, they propose the classes of deictic, gesticulation,
manipulation, semaphores, and sign language. In thismodel, gesticulation is also equated with iconic or depictive
gestures, which are used to depict physical shapes andforms referred to by speech.
Our work adopts elements of where appropriate to build ourclassification system, primarily by extending Karam andschraefel’s model. We also build on other work in HCI.

Gestures in HCIMany novel interfaces, such as those for in-air gesturedetection [2, 3], multitouch [22], tangible [14], and organic[18] user interfaces try to overcome limitations of WIMP
interaction. Interfaces combining hand-gesture and speech
try to provide means of control by mimicking human-tohuman communication [4, 24].
In the present work, we are focusing on purely gesturedriven human computer interfaces, without coverbal
gesticulation. A notable demonstration of unimodal mid-air
gesture usage is found in the demos for the g-speak
platform (oblong.com). The designers show several sample
applications for interacting with virtual environments,
including drawing, image and video editing, and navigationin 3D space. They use bimanual interaction extensively,
primarily to establish reference frames. This work extends
earlier work in HCI, such as Charade [2], as well as work
specifically dedicated to the creation of gesture languages.
Creating Gesture Languages

The study of the construction of gesture languages can beroughly divided into two camps: one viewpoint is thatgesture languages must be elicited, through the study ofpotential users in their natural environments, or throughuser definition [20, 21, 28]. This approach prioritizes theprobability that a new users’ first guess of a gesture will
yield the desired effect. The alternative camp proposes thatgesture languages must be designed through an examinationof the unique physical and psychological implications of agiven input device and context of its use, and then taught tothe user [1, 9, 27]. Their view is that ‘guessability’ may be
only one of several factors in the design of a language.
With our work we attempt to bridge the gap between thesecamps. By building on UDI methodology, we are seeking
for typical user behaviors in terms of hand usage forconstrained communication. These behaviors, however, arein the form of types, rather than particular gestures.
Through the definition of these classes, we seek to provide
generally useful guidance on the issue of ‘guessability’ tothose practitioners and researchers who are considering thisand several other factors in designing gestural languages.
In seeking gesture types, we follow the general approach
proposed by Nielsen et al. [21]. They suggest that in order tobuild a gesture interface it is necessary to first identify thefunctions that will be evoked (or ‘effects’ of the gestures).
Subsequently, the most appropriate gesture for each of thosefunctions has to be found (in our case, gesture types). In our
methodology, we select 10 common functions (‘effects’)required in many interactive systems, and elicit and classify
gesture performed by our participants into gesture types.
TARGET GESTURE EFFECTS

A difficulty in defining a classification of gesture types isthe need to develop generic and representative gestureeffects (system or application functions). Foley et al. [7]identified six generic tasks, reflecting the user’s intentions:select position and orient an object, ink (draw lines), enter

text, and specify scalar values. This work is somewhat
limited for our purpose, because it assumes mouse-likeprimitives. For example, it assumes that possible effects
such as accepting or refusing will be composed of otherprimitives, such as selection of a menu item, by selecting(activating) a button, or by entering text.
To achieve coverage of frequently desired gesture effects,
we attempted to find as many effects as possible: during a
brainstorming session with four researchers, we gathered alarge set of tasks and actions for content creation,
manipulation, management, and navigation in the physicalworld. We do not claim the resulting list to be complete, forexample it excludes tasks for heavy data input, such as text
input. However it should cover most actions one would liketo perform with a gesture interface. Next, we split highlevel actions into atomic physical acts, in order to find abasic set of effects. For example, the task of arranging
several blocks by their color can be decomposed into
several acts of moving an object, which in turn can be split
into grasping, translating and releasing. This is similar tothe logical primitives approach taken by Foley et al., exceptthat our primitives are based on physical acts, rather than anassumed interaction model.
From this process, we formed a list of 10 fundamental, nonoverlapping effects:
 Select: in contrast to identification, selecting refers to
the actual action of picking or grasping of physicalentities.
 Release: this effect is the complement to selecting, thus
it expresses the depositing of previously selected
entities.
 Accept: this represents agreeing in binary queries andincludes similar meanings such as approving andaffirming, as well as continue in the sense of “you aredoing it right, go on”.
 Refuse: the complement effect to accepting is often used
for answering “no” in binary queries, but it also includes“wrong”, “stop”, or “undo”.
 Remove: in contrast to releasing, removing entities
means to dismiss them, to erase them, or even to movethem off the table.
 Cancel: this effect represents pausing or stopping of acurrently ongoing process, or even terminating thewhole overall task.
 Navigate: in contrast to translation, navigation refers to
the guiding (both movement and rotation) of arepresentation of the actor’s hand or tool, such as a pen.
This representation is considered to be similar to amouse cursor.
 Identify: this effect includes identification, indication or
description of an entity or location.

 Translate: other than navigation, translation refers to the
movement or the dragging of a physical entity.
 Rotate: this effect represents the action of rotation ofphysical entities.
Having defined the atomic effects of gestures we wished toevaluate, we turn our attention now to the design of ourexperiment, which was meant to answer the question: whattypes of gestures do users perform in order to achieve theseeffects? We extended the UDI methodology in order tolearn not only ‘first guess’ behavior, but to record theresults of refinement of techniques over time – given thepresence of a closed feedback loop.
EXPERIMENT FOR COLLECTING GESTURE DATA

The goal of this study was to determine mappings of userdefined gesture types and number of hands used to each of
the effects. To do this, we constructed tasks which, in orderto accomplish them, would include the 10 effects we
identified. The experiment was done in pairs of two: onewas designated the actor, who performed gestures, and thewizard, who observed the gestures and carried out theeffects that they understood. The wizard was provided withthe tools needed to complete each task, but was not madeaware of the goals, or which of the tools would be required
to accomplish them. There was no a priori communication
about the tasks between the wizard and actor, nor were any
guidelines provided to shape their communications. The
wizard was not given aid to understand what task would be
performed, beyond the placement of the necessary tools
within their reach.

meant to be one-way, thus from actor to wizard, while thedesired closed feedback loop was established by showing
the wizard’s actions to the actor. Both participants wereplaced in front of a camera/screen setup, as shown in Figure
2. The actor was only able to see the table, all tools, and thewizard’s hands; the wizard was shown both the upper bodyand the head of the actor. In order to help the wizards tolocate their work, the borders of the area visible to the actor
were marked on the table.
For each task, the actor was provided with the goal of theresults to be achieved. Depending on the task, this waseither a drawing or a photograph. The image was placednext to the actor’s screen throughout the task.
Participants had to finish each task within 10 minutes; afterthat time, the task was suspended. Since we intended tocollect as many gestures as possible, we designed the tasks
in a way that prevented their accomplishment during theallotted time. All participants were told not to hurry, tocommunicate at a convenient speed, and simply try tocomplete as much of each task as possible.

Participants

Twelve participants (8 male), with ages ranging from 23 to
54 years (M = 30.92, SD = 10.21) were recruited from a
Central European community. To avoid technical bias, wechose candidates with affiliations ranging from school
teacher, lawyer, and process manager to IT technician andsoftware developers. Mean self-reported daily computer
usage was 6.07 hours (SD = 3.08). 7 participants reported
frequent use of gesture based touch interfaces for a duration
ranging from 4 months to 4 years (M = 2.17, SD = 1.59),
such as smartphones and tablets, 6 of those also had used
mid-air gesture interfaces (3 of them frequently) prior to thestudy, such as Microsoft Kinect, WiiMote and PlayStation
Move. The remaining 5 did not have mentionable
experience with gesture interfaces whatsoever. The
participants were divided into 6 groups of 2 strangers. To
eliminate biasing by gesture vocabularies established during
the earlier trials, the task order was counterbalanced using a
6×6 Latin Square.
DesignBoth actor and wizard were placed in separate rooms toprevent them speaking or using other forms of auditorycommunication. Both participants were connected by avideo chat system (Skype), with the audio channel disabled,so the only reasonable high-bandwidth channel left was
gestures. In our experiment gesture communication was

Figure 2. In our study, 2 participants, the actor and the wizard,
were separated spatially, and were only allowed visualcommunication using video conferencing.

After each task, actor and wizard switched roles and moved
on to the next task. This means that each participant acted
as both actor and wizard, providing greater variety ofgesture types for our analysis.
Tasks

We chose the 6 tasks to be completed during the study.
Since our focus is on gesture interfaces for HCI they were
not chosen to be particular meaningful everyday humanhuman communication tasks but to cover a majority of lowlevel action, thus our 10 chosen effects. For example
building a figure out of toy blocks would at least require (i)identification of an object, or alternatively navigation of the
wizard’s hand, (ii) grasping, (iii) translation and rotation of
the object, and (iv) releasing. Samples of the goal images
for each of the 6 types of tasks are shown in Figure 3.
Pilot studies aided in refinement of the tasks to fine-tune
their difficulty and diversity. The selected six tasks were: Blocks: the objective was to reconstruct a given figureout of DUPLO blocks (Figure 3a), and to dismantle theconstruction again after 9 minutes.

 Drawing: the goal was to draw a number of figures ontoa blank piece of paper (Figure 3b). After 6 minutes, theactor had to instruct the wizard to memorize everythingdrawn so far and to redraw it on a new sheet of paper.
This was included to find gestures for the effect save.

(a)

(c)

GESTURE TYPE CLASSIFICATION SCHEME

In a pilot study, we used the taxonomy of Karam andschraefel [15]. As we conducted the coding of the video, we
found the need to modify Karam & schraefel’s scheme. In
particular, it was clear that their gesticulation class failed tocapture the distinction between iconic and pantomimic
gestures, and so was split into those two categories. We
further observed iconic gestures, like semaphoric in theearlier scheme, could be further divided into static anddynamic, though stroke did not apply.

(b)

As formal sign languages have to be learned just like any
spoken language, we did not expect them to appear during
the study, so we omitted them in our classification scheme.
Since they are especially designed to be versatile and highbandwidth, they are rather complicated and inconvenient touse and learn [21]. As a result, we find them to be unsuitablefor guessable and intuitive gestures.

(d)Additionally, we replaced the term deictic by pointing, since
our application did not use speech communication, and thus
there can be no deixis per se. Our classification scheme,
which is a modified version of the previous work, is shown in
Figure 4. We now explain each of the gesture types in greater
detail, and then present the results of the coding exercise.
Pointing(e)(f)

Figure 3. Samples of the goal images provided toeach actor, and hidden from each wizard.

 Maze: the actor had to make the wizard draw a series ofpreselected paths through a maze (Figure 3c).
 Meter: six glasses with liquid were to be filled to
specified levels and in a specified order. There wereseveral intermediate states to accomplish (Figure 3d).
 Arrange: the objective was to recreate a specificarrangement of colored cutout arrow shapes, as depicted
on a printout presented to the actor (Figure 3e).
 Sculpt: wizard was to recreate an abstract model out of
clay, as depicted in a set of photographs provided to theactor (Figure 3f).
To avoid agency by the wizard, we incorporated aspects into
each task which were unlikely to be anticipated, such as
moving backwards in the maze task, drawing a fish onto atree, and dismantling the incomplete block figure.
We recorded both the actor’s and the wizard’s video stream
and used them for our analysis. Twelve participants
directed three tasks of ten minutes each, which resulted in
six hours of video data. In order to classify the gesture
types, we developed the following classification scheme.

Pointing is used to indicate objects and directions, which
does not necessarily involve a stretched index finger. It may
also be performed with multiple fingers, the thumb, a flat
palm, etc. Note that gestures which resemble pointing but
which are intended to mean “yes, you understand” werelabeled as semaphoric, given their loaded meaning.
Semaphoric

Semaphoric gestures are hand postures and movements,
which are used to convey specific meanings. Mostly gestureand meaning are completely unrelated and strictly learned.
Therefore, we consider semaphorics to be the gestures mostdependent on the actor’s background and experience.
Static semaphorics are identified by a specific hand posture.
Examples would be a thumbs-up, meaning “okay”, or a flat
palm facing from the actor, meaning “stop”.
Dynamic semaphorics convey information through theirtemporal aspects. A circular hand motion meaning “rotate”
is of this type, as well as a repeatedly flicking or waving of
the hand sideward, meaning “no”.
Semaphoric strokes represent hand flicks and are similar to
dynamics, since they are identified only by hand motion,
but which are single, stroke-like movements. They may becompared to the familiar touch and stylus gestures on iOS
and Windows. An example would be a single, dedicatedsideward flick of the hand, meaning “dismiss this object”.
Pantomimic

Pantomimic gestures are used to demonstrate a specific task
to be performed or imitated, which mostly involves motion
and particular hand postures. They usually performed by anactor without any objects actually being present, such as
filling an imaginary glass with water, by tilting an imaginary
bucket. They often consist of multiple low-level gestures,e.g., (i) grabbing an object, (ii) moving it, and (iii) releasing
it again. We code these as a single pantomimic gesture.

gestures is the presence of the feedback loop. In the case ofmanipulation gestures, the actor waits for the entity to“follow” before continuing, instead of performing
beforehand, only causing a reaction subsequently.
RESULTSTo review the results of our study, we begin with classifying
each of the observed gestures used for each effect.
Gesture Effects

Three researchers collected approximately 5,500 gesturesand categorized them using our classification scheme,
constantly consulting each other in order to prevent frombiasing and diverging interpretations. We also differentiatedby unimanual and bimanual gestures. The overall results are
depicted in Figure 5. This represents our primarycontribution: the types of gesture chosen for each of thedesired effects.
Note that some gestures may include elements of more than
one type, as actors displayed diverse creativity incomposing gestures from different types. This is especiallyapparent in the case for bimanual gestures, since gestures
might be combined, such as expressing “move the roundblock” by forming a round static-iconic gesture and thenpantomiming a movement with that shape. Also, somemovements were not gestures to convey meaning, such as
when the actors hesitated or when they were irritated. As aresult, the sums for any given effect may not add up to100%.

Figure 4. The classification we used to analyze gestures in thisresearch, including examples for each of the gesture types.
IconicIconic gestures are used to communicate information about
objects or entities, such as specific sizes, shapes, andmotion paths.
Static iconics are performed by static hand postures. In
contrast to static semaphorics they do not rely on acommonly known vocabulary, instead they are rather
spontaneous, such as forming an “O” with index finger andthumb, meaning “circle”.
Dynamic iconics are often used to describe paths or shapes,
such as moving the hand in circles, meaning “the circle”.
Compared to concatenated flicks (which would besemaphoric strokes), the motions are usually performed
more slowly. Another difference is that in strokes, theactual range of the movement does not hold information
about the action, however in dynamic iconics it does.
Manipulation Gestures

Manipulation gestures are used to guide movement in ashort feedback loop. Thus, they feature a tight relationshipbetween the movements of the actor and the movements of
the object to be manipulated. The criterion for
distinguishing them from pantomimic and dynamic iconic

As predicted, actors used a wide variety of gestures toaccomplish the same effect. However, the type of gesturethat they used was often consistent across time, andparticipant. Thus, while a classification of particular
gestures might find a high degree of variance [28], ourresults suggest that classifying by type reveals a muchgreater degree of consistency.
SelectSelection was most often indicated with pantomimic
gestures, primarily in the form of “grasping”. Bimanual
gestures were rarely present for selection.
ReleasePantomimics, semaphoric strokes and iconic dynamics
showed high proportions of bimanual acting. Pantomimic
gestures were “releasing hand gestures”, thus thecounterpart of the grasping gestures used for selection.
Semaphoric strokes were usually flicks downwards, mostly
with stretched palms facing down, indicating placement of
the object down onto the table.
AcceptAll gestures were semaphoric static, either by showing
thumbs-up hand poses, okay-signs (foming an “o” withindex finger and thumb), or the previously described
pointing-like semaphoric gesture.
Refuse97% were semaphoric dynamic gestures, either by waving
sideward with one or two hands and palms facing down,while index finger or palm were outstretched, or by waving
sideward with the index finger pointing up.
RemoveSemaphoric strokes were sideward flicks. Bimanual strokes
mainly showed both hands flicking into the same direction.
Unimanual pantomimics were mostly throw-away-gestures,e.g., showing the actor throwing an imaginary object
backwards, over the shoulder. The high bimanual portion
was mainly made up of put-aside-gestures, i.e. grasping aninvisible object with both hands and releasing it offside.

arrange task the tip of an arrow was sometimes indicatedby an outstretched index finger or thumb. Pantomimic
gestures simply showed the rotations to be performed by a
combination of grasping, rotating, and releasing. Most of
the semaphoric dynamic gestures were circular motions of
the hand, with the index finger or even the palm
outstretched. The bimanual portion represents flip-gestures,with both arms moving in a half circle, ending up in crossed
arms, referring to a rotation of 180°.
SelectReleaseCancel60%

60%

Strokes were generally flicks sideward, with outstretchedpalms facing down and both hands moved into opposite
directions. Semaphoric statics mostly showed hands withoutstretched palms facing away from the actor (“stop”
gesture). Okay-signs and thumbs-up hand poses weresometimes observed to indicate “Alright, good job, stopnow.” Semaphoric dynamics were variations of the strokes
mentioned above, but with the hands repeatedly waving,instead of flicking once.

40%

40%

20%

20%

Navigate

Navigation was mainly done by pointing into the desired
direction, which was mostly accompanied with a rhythmic
movement. Manipulation gestures guided the wizard’s
hand as if it were a remote representation of the actor’shand, comparable to a mouse cursor. Semaphoric strokes
were used to indicate the direction to move towards.
Bimanual gestures were rarely present.

0%

0%
Pan

Translate

Semaphoric dynamics were mostly repetitious waving
towards the respective direction, with one or both hands.
Pointing gestures were mostly rhythmic pointing into thedesired directions; this was generally done with one hand.
About 10% out of those included self-references, thus using
one hand or arm, or even other parts of the body to establisha reference frame. Pantomimic gestures were
demonstrations of the translations, including hand gestures
mimicking grasping and releasing. In manipulation
gestures, actors guided the wizard using slow movements,
while the wizard followed accordingly
RotateIconic dynamics were performed by according rotations of
the hand, mostly with an outstretched palm, while the hand
was considered a representation of the object. In the

S.Str S.Sta Others

Pan

S.Str I.Dyn S.Dyn Others

AcceptRefuse

100%

100%

50%

50%
0%

0%
S.Sta

S.Dyn

OthersRemoveOthersCancel

40%
30%
20%
10%
0%

60%
40%
20%

0%
S.Str

Identify

While pointing seems to be obvious for identification,
iconic dynamic gestures were used to describe the desired
object by drawing its shape in mid-air. Semaphoric statics
were used to identify objects by showing a number offingers. Identification like this requires the priorestablishment of a code; actors would occasionally index
objects by numbering them at the beginning of the task.
Iconic static gestures were used to refer to objects byshowing their size.

Ptg

Pan

S.Str

S.Dyn

Navigate

S.Sta

S.Dyn Others

Identify

80%
60%
40%
20%
0%

60%
40%
20%

0%
Ptg

Man

S.Str

Others

Ptg

Translate

I.Dyn S.Sta I.Sta Others
Rotate

40%
30%
20%
10%
0%

40%
30%
20%
10%
0%
S.Dyn Ptg

Pan

Man Others

I.Dyn Pan S.Dyn Ptg Others

Ptg - Pointing
S.Str - Semaphoric strokes
Pan - Pantomimic
I.Sta - Iconic static
S.Sta - Semaphoric static
I.Dyn - Iconic dynamic
S.Dyn - Semaphoric dynamic Man - Manipulation

bimanual
unimanual

Figure 5. The type histograms for each of the effects.

As intended by the study designers, gesture communication
was one-way for most the time. Gestures performed by the
wizards, e.g. due to confusions, were insignificant, although
the participants were not explicitly advised to refrain.
Feedback was mostly given by the effect, visible on screen.
Influence of Participant’s Experience

For select, remove, translate, and rotate, significant
differences between groups were observed depending on

the participant’s prior experience with gesture interfaces.
For the rest of the effects, there are only minor differences:
there was a relatively low change within type usage(average 4.27%, SD = 0.0372), while for each effect, thetypes with the highest portion were the same for bothnovices and experts, and the second most prominent types
only changed in navigate and deactivate.
Reference Frames

The usage of reference frames, such as self-references, isshown in Figure 6. They are especially present in iconic
static, iconic dynamic and pantomimic gestures, with anotably bimanual dominance, validating previous design
efforts [11]. Reference frames using one hand usually
involve prior established contexts, such as drawing a
rectangle into mid-air with the outstretched finger,
representing the drawing canvas, followed by a circular
movement in the upper left corner, meaning “draw a circleof this size into this corner of the sheet.”Ptg - Pointing
S.Str - Semaphoric strokes
Pan - Pantomimic
I.Sta - Iconic static
S.Sta - Semaphoric static
I.Dyn - Iconic dynamic
S.Dyn - Semaphoric dynamic Man - Manipulation

Man

I.Dyn

I.Sta

S.Str

S.Dyn

S.Sta

Pan

Ptg

30.00%
25.00%
20.00%
15.00%
10.00%
5.00%
0.00%

bimanual
unimanual

Figure 6. Reference frame usage for each of the types.

It was our goal to record results in the presence of a
feedback loop, and without the introduction of a recognizer
which would artificially affect results. Nonetheless, that the
wizard was a human observer has an obvious impact on our
measurements: vocabularies which were easily understoodby the counterpart became the most present overall. Some
effects saw gesture types change after a short period, as theactor adapted to the preference of the wizard. This wasmost prominent for translate, rotate, remove, and cancel,
whereas accept and refuse did not show significant changes
over time. While this grounding process did not introduce
significant error into our data, since failing gestures were
dropped instantly, it does have an impact on how tointerpret our results. Thus, they do not rely on the most
spontaneous gestures for each action, but on the mostcommon instead. It remains an open question as to whether
the grounding process would be similar whencommunicating with an artificial gesture learning system.
The degree to which this shift occurred varied acrossdesired effect - the remaining six effects were less affectedby this shift. In particular, accept and refuse did not show
significant changes of gesture type over time.

DISCUSSION AND DESIGN IMPLICATIONS

In general, participants displayed a lot of unanticipatedbehavior. For example, we expected manipulation gestures
to be much more present, especially for translation, rotationand navigation. Instead, participants tended to iconicallydescribe motion paths and object shapes, and todemonstrate rotations beforehand. In latter case, a pointing
finger seems to be a good way of clarifying the exact angleof rotation whenever the rotated object had a clear front andback side, just like the arrows in the arrange-task. If theobject did not have such clear distinctions of front and backsides, a pointing finger might still be useful as an “anchor”.
These results have significant implications for the design ofalternative implementations of “direct manipulation”.
Some of the results were highly consistent, especially those
for accept and refuse. Other results showed much morevariety, which suggests that there is more flexibility ingesture language design for those cases. For example, forremove, the three types of semaphoric strokes, pantomimics
and semaphoric dynamics were performed at similar rates.
This suggests that those types are equally suitable.
The fact that the types for accept and refuse were distinctly
different is particularly interesting, because these two
effects are dichotomous. The static hand postures for
accepting are completely opposed to the dynamic handmotions used for refusing. This is highly valuable since
confusing dichotomous effects would be particularly
severe, yet static and dynamic motions can easily be
distinguished by a gesture recognition system.
Another set of dichotomous effects is select and release.
While both were mostly accomplished by pantomimic
gestures, bimanual acting was highly present in the latter,
which could be valuable for distinguishing between them.
In contrast to accept/refuse, it would even be possible todistinguish between them by context in many applications.
For identification of objects, pointing is often ambiguous.
In applications with many objects located close to eachother, it might be valuable to describe objects iconicallyinstead. Alternatively, pointing can also be used to preselecta number of possible candidates (the ones close to the spot
the user is pointing to). Those could be augmented, withindices, so the user is able to select the desired object byshowing the corresponding number with a semaphoric static
gesture, for example by stretching out three fingers to for“object #3”. It is important to note that this gesture type was
employed despite each participant also discovering the
cursor-like ‘manipulation’ gestures, suggesting a potential
preference vs. traditional ‘cursor’ pointing.
Although there is little consensus between users regarding
association between specific gestures and effects, there
seem to be some exceptions. One gesture was particularly
prevalent throughout all participants: all actors pointed one
or both index fingers to their head meaning, “memorize thedrawing” in the drawing task.

The processes of translation and navigation seem to be very
similar at first sight since both effects refer to movements:
those of an object to be manipulated in the former, andthose of an avatar or cursor in the latter. By comparing theresults, however, we can see that there are significant
differences in type. While translation is mostly
accomplished by semaphoric dynamic gestures, navigationis generally done by pointing. This indicates that there is a
substantial difference in how to instruct movements,
depending on the relation between actor and entity.
Actions for object manipulation (translations and rotations)
are mostly accomplished using hand motion, thus bydynamic gestures. However, translation is done bysemaphorics, e.g., by repetitious waving, which has to be
explicitly terminated when the target is reached. In contrast,
rotation already shows one concluded movement, e.g., byrotating the palm around 45°, which already contains all theinformation required for performing the action frominitialization of the action to its conclusion.
Pointing is highly present in both navigation andidentification types. While this might indicate danger ofconfusion on first sight, identification can mostly be
distinguished from navigation by watching the hand pose:
while the actor points directly at the objects (thus on thescreen) in the former, s/he points sideward (left, right, up,down) in the latter. Also the context (e.g., an application’s
state) might be an indicator in many cases.
Semaphoric strokes are mainly used for removing andcancelling. While there are other types in removing, which
are almost as frequent, the result for cancelling is much
more distinct, and is highly bimanual.
Overall, the extent of bimanual gesticulation is clearly
noticeable. This implies that it is appropriate to extensively
incorporate it into HCI interfaces. Bimanual gestures arehighly popular in pantomimic and iconic gesture types, withthe exception of those used for the effect of navigation andselection.
Limitations of this WorkA potential limitation of this work is possible dependence
on culture. So far, we did not compare actors acrossdifferent countries and cultures. While particular gesture
selections may vary across cultures, we hypothesize that
only minor differences in gesture type would be observed.
E.g, our central-European actors frequently made ‘stop’
gestures by facing the palm of a single hand towards the
wizard. The Japanese equivalent for that effect is crossing
both arms in front of the upper body. In both cases, the typeis semaphoric static.
While using a human recognizer instead of a gesturerecognition system may have led to a more unconstrained
behavior of the actors, it is not clear yet to what extent thisintroduced characteristics of human-human dialogue andhow they may affect our findings. While we argue that it
would be reasonable to model these in a designed gesturelanguage, more research has to be done in order to judge
their respective applicability in HCI.
Another limitation is the availability of additional
communication channels to the wizards, especially facial
expressions and body postures. Although they already werepart of the communication during the pilot study, weobserved the overwhelming majority of them to be used
only for emphasis, rather than to convey independent or
additional meaning. As a result, we found them to becompletely redundant information channels, so we did notintend to hide them from the wizards.
Although we designed our tasks to reduce the opportunity
for agency of the wizard to interfere, some did occur.
However, we did not find that asymmetrically benefited any
one gesture effect or type.
CONCLUSION & FUTURE WORKIn this paper, we presented the design of a user study,
which extended UID, a new classification system for midair gestures, and observations regarding gesture type usage
for ten different gesture effects.
It is interesting to note that, as of today, many systems arebuilt upon the metaphor of direct manipulation while ourresults point out that the interfaces may better map to users’
expectations if they were able to recognize other types ofgestures for each action. Our results can serve as a guide,for both designers and researchers, to the type of gestures
which are appropriate for various effects.
In particular, our results show that users tend not to use
manipulation gestures to translate or rotate physical objects,
while selecting and releasing is primarily done
pantomimicly. There are huge differences in terms of
motion for accepting and refusing. Bimanual acting ishighly present, although only for certain combinations ofgesture types and effects.
The most direct application of these results will be thedevelopment of alternative gesture languages, taking into
account fundamentals of gestural communication. It is ourhope that designers will begin to design gesture languages
based not only on those mappings which are most‘guessable’ by a novice, but instead taking into account the
many factors of context which affect the definition of
gestures. Our results can serve as a guide to those designers,
since it frees them from UDI’s simplistic 1:1 mapping ofparticular gestures to particular effects, and instead provides
categories of gesture types from which they can draw.
ACKNOWLEDGEMENTS

The authors gratefully acknowledge that the work for thispaper was funded by the FFG on behalf of the Austrian
Government as part of the research project CADET.

REFERENCES

1. Apitz, G., and Guimbretière, F., CrossY: a crossingbased drawing application. In Proc. UIST 2004, ACM
Press (2004), 3-12.
2. Baudel, T., and Beaudouin-Lafon, M., Charade: remotecontrol of objects using free-hand gestures. In Commun.
ACM 36, 7, ACM Press (1993), 28-35.
3. Benko, H., and Wilson, A.D., Multi-point interaction
with immersive omnidirectional visualizations in adome. In Proc. ITS 2010, ACM Press (2010), 19-28.
4. Bolt, R., “Put-that-there”: Voice and gesture at thegraphics interface. In Proc. 7th Annual Conference on
Computer Graphics, ACM Press (1980), 262-270.
5. Cadoz, C., Les réalités virtuelles. Flammarion, Paris,
1994. ISBN: 2-08-035142-7.
6. Efron, D., Gesture and environment. The Hague:
Mouton, 1972.
7. Foley, J.D., Wallace, V.L., and Chan, P., The humanfactors of computer graphics interaction techniques. In
IEEE computer graphics and applications 4, 11, IEEE
Computer Society Press (1984), 13-48.
8. Forlines, C., Wigdor, D., Shen, C., and Balakrishnan, R.
2007. Direct-touch vs. mouse input for Tabletop
displays. In Proc. of CHI '07, 647-656.
9. Freeman, D., Benko, H., Morris, M.R., and Wigdor,D.,
ShadowGuides: visualizations for in-situ learning ofmulti-touch and whole-hand gestures. In Proc. ITS
2009, ACM Press (2009), 165-172.
10. Fussell, S.R., Setlock, L.D., Yang, J., Ou, J., Mauer, E.,
Gestures over video streams to support remotecollaboration on physical tasks. In Human-Computer
Interaction 19, L. Erlbaum Associates Inc. (2004), 273309.
11. Gustafson, S., Bierwirth, D., and Baudisch, P. 2010.
Imaginary interfaces: spatial interaction with empty
hands and without visual feedback. In Proc. of UIST '10.
3-12.
12. Hilliges, O., Izadi, S., Wilson, A.D., Hodges, S., GarciaMendoza, A., and Butz, A., Interactions in the Air:
Adding Further Depth to Interactive Tabletops. In Proc.
UIST '09, ACM Press (2009), 139-148.
13. Izadi, S., Hodges, S., Taylor, S., Rosenfeld, D., Villar,N., Butler, A., and Westhues, J. Going beyond thedisplay: a surface technology with an electronicallyswitchable diffuser. In Proc. UIST '08. ACM Press(2008), 269-278.
14. Jordà, S., Geiger, G., Alonso, M., and Kaltenbrunner,
M., The reacTable: exploring the synergy between live
music performance and tabletop tangible interfaces. In
Proc. TEI 2007, ACM Press (2007), 139-146.
15. Karam, M. and schraefel, m.c. (2005) A Taxonomy ofGestures in Human Computer Interactions. Technical

Report ECSTR-IAM05-009, Electronics and Computer
Science, University of Southampton.
16. Kendon, A., How gestures can become like words. In F.
Poyatos (Ed.), Crosscultural perspectives in nonverbal
communication. Hogrefe, 1988, 131-141.
17. Kirk, D., Crabtree, A., Rodden, T., Ways of the hands.
In Proc. ECSCW ’05, Springer-Verlag New York, Inc.
(2005), 1-21.
18. Koh, J.T.K.V., Karunanayaka, K., Sepulveda, J.,
Tharakan, M.J., Krishnan, M., and Cheok, A.D., Liquidinterface: a malleable, transient, direct-touch interface.
In Comput. Entertain. 9, 2, ACM Press (2011), 7:1-7:8.
19. McNeill, D., Hand and Mind: What Gestures Reveal
About Thought. The University of Chicago Press,
Chicago, London, 1995. ISBN: 0-226-56134-8.
20. Morris, M.R., Wobbrock, J., and Wilson, A.
Understanding Users’ Preferences for Surface Gestures.
Proceedings of Graphics Interface 2010, 261-268.
21. Nielsen, M., Störring, M., Moeslund, T.B., and Granum,E., A procedure for developing intuitive and ergonomic
gesture interfaces for HCI. In: Camurri, A., and Volpe,G., (Eds.) Gesture Workshop 2003. LNCS (LNAI) 2915.
Springer, Heidelberg, 2004, 409-420.
22. Reisman, J.L., Davidson, P.L., Han, J.Y., A screenspace formulation for 2D and 3D direct manipulation. In
Proc. UIST 2009, ACM Press (2009), 69-78.
23. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T.,
Finocchio, M., Moore, R., Kipman, A., and Blake, A.,
Real-time human pose recognition in parts from single
depth images. In Proc. CVPR 2011, IEEE (2011), 12971304.
24. Stern, H.I., Wachs, J.P., and Edan, Y., Designing hand
gesture vocabularies for natural interaction by
combining psycho-physiological and recognition
factors. In Int. J. Semantic Computing 2, 1 (2008), 137160.
25. Vogel, D., and Balakrishnan, R., Distant freehand
pointing and clicking on very large, high resolution
displays. In Proc. UIST 2005, ACM Press (2005), 3342.
26. Wang, R.Y., and Propović, J., Real-time hand tracking
with a color glove. In ACM Trans. Graph. 28, 3, ACM
Press (2009), 63:1-63:8.
27. Wigdor, D., and Wixon, D., Brave NUI World:
Designing Natural User Interfaces for Touch andGesture. Morgan Kaufmann, 2011. ISBN: 978-0-12382231-4.
28. Wobbrock, J.O., Morris, M.R. and Wilson, A.D., Userdefined gestures for surface computing. In Proc. CHI
2009, ACM Press (2009), 1083-1092.