Understanding Mid-Air Hand Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI

In this paper we present the results of a study of human preferences in using mid-air gestures for directing other humans. Rather than contributing a specific set of gestures, we contribute a set of gesture types, which together make a set of the core actions needed to complete any of our six chosen tasks in the domain of human-to-human gestural communication without the speech channel. We observed 12 participants, cooperating to accomplish different tasks only using hand gestures to communicate. We analyzed 5,500 gestures in terms of hand usage and gesture type, using a novel classification scheme which combines three existing taxonomies in order to better capture this interaction space. Our findings indicate that, depending on the meaning of the gesture, there is preference in the usage of gesture types, such as pointing, pantomimic acting, direct manipulation, semaphoric, or iconic gestures. These results can be used as guidelines to design purely gesture driven interfaces for interactive environments and surfaces.  Recent advances in computer vision, particularly in realtime hand [26] and body tracking [23] are empowering computers and intelligent environments to recognize human gestures from a distance. These novel technologies reduce barriers to interaction and increase the input bandwidth between the user and the computer, without requiring the user to wear or acquire a tracked object. Since surface and tabletop systems have expanded to include proximity sensing [13], there is potential for computer vision based mid-air interaction to become a part of tabletop computing [12] and large-scale displays . The recognition of mid-air gestures enables designers to create interfaces that enable explicit control of such systems. As is true of any new input modality, there is a need for extensive design of the physical primitives which make-up the interaction . Since the WIMP GUI is still a prevalent metaphor in computer interfaces, a simple approach is to adopt its “point and click” and related primitives. For example, Vogel and Balakrishnan  built a pointing-andclick interface for large screen displays which simulates mouse input. They identified issues with precision in pointing, ambiguity of finger movements, and lack of physical feedback. These issues are critical for the control of a point-and-click, mouse-based UI, but offer little guidance to designers wishing to develop new, possibly more suitable input primitives. In particular such transformation may reduce much of the richness of the new input modality, and sacrifice available bandwidth . To create interfaces that leverage a gestural input paradigm, there is a need to build gestural interfaces from scratch – including the design of new physical primitives. One common approach has been to elicit gestures applying the user defined interfaces (UDI) methodology. This technique is useful in determining a novice’s “first guess”. However, researchers have found that there is little consensus among users in association between gestures and their expected effect [24, 28]. In contrast, designed languages can be formed to avoid conflicts, but also to be ergonomic and high-bandwidth. In view of this, we believe that gesture languages need to be designed, rather than observed. A key Copyright Microsoft Corporation Microsoft Research Technical Report MSR-TR-2012-111. limitation, however, is the lack of guidelines to aid designers in creating those languages. To overcome this issue, we seek to bridge the gap between those two methodologies by investigating what linguists term gesture types . Although gesticulation humans perform during speech is quite different in its nature, we build on this traditional linguistic practice for a more systematic investigation of gestures in HCI. Rather than seeking to construct a dictionary of particular gestures for a given set of functions, classifying physical actions enables us to create middle ground between elicited and designed gestures. While a UDI methodology might provide viable but inconsistent results, our approach enables for a thorough design of a gesture language by providing guidance that, for example, a gesture for a particular action should be chosen to involve unimanual static hand postures or bimanual movements, or that it should use pointing, descriptions of the objects, or semaphoric codes. In order to produce our gesture types, we conducted a study. Our methodology was similar to that of UDI, in that participants were given a ‘before’ and ‘after’ situation, where user-selected gestures were used to make the transition between those two states [28]. Unlike previous work, our study provided a recognizer – a second participant, who was observing, via a muted video feed, the actions of the first participant. The first participant was tasked with guiding the observer in completing various tasks, using only gestures shown over a video feed , thus lacking speech. The ‘human’ recognizer enabled participants in our study to perform gestural interaction within a closed feedback loop, enabling refinement over time. The choice of using a human instead of an artificial gesture recognizer was also taken to prevent from biased behavior and to provide an unconstrained environment to the directing participant. While this study setup is similar to previous works in CSCW [10, 17], the goals and information collected were different. In our methodology the actor was meant to communicate only using mid-air gestures, in order to accomplish a task in a ‘virtual’ environment, just like in controlling an interactive computing system.. Except for visual feedback showing the result of the given input, the communication was intentionally one-way, as with a unimodal gesture interface. Additionally, the study was designed not to address highlevel human communication tasks, but low-level actions such as manipulating objects, guiding avatars, and describing simple shapes, in order to collect gesture data for each of them for further analysis. As we will discuss, the refinement process saw the introduction of new types of gestures, ad hoc formalisms were created, and whole gesture languages were observed. During this process, we collected approximately 5,500 gestures from 12 participants during 6 different scenarios. We analyzed those in terms of expected effect and gesture type, using our own classification scheme. We also collected data about the number of hands used, by splitting into unimanual and bimanual gestures. The results indicate that for each expected effect of a gesture, there are preferences in both type and number of hands. These preferences, reported in detail in this paper, represent two contributions. First, they provide a baseline measurement of conventional human behavior (after a short grounding process) which goes beyond the ‘first guess’ provided by UDI. Second, the results, in and of themselves, form the basis for the development of guidelines which designers can use to design their own gesture languages, taking into account not only user preference, but all the factors relevant to the context of the language . In this paper, we present the design of the study, our classification system, and our findings about gesture type usage, including user choices for univs. bimanual gestures. Our contribution is our classification scheme, and knowledge of the suitability of gesture types for the chosen gesture effects.