# A Taxonomy of Microinteractions: Defining Microgestures Based on Ergonomic and Scenario-Dependent Requirements

Katrin Wolf, Anja Naumann, Michael Rohs, Jörg Müller

► To cite this version:

Katrin Wolf, Anja Naumann, Michael Rohs, Jörg Müller. A Taxonomy of Microinteractions: Defining Microgestures Based on Ergonomic and Scenario-Dependent Requirements. 13th International Conference on Human-Computer Interaction (INTERACT), Sep 2011, Lisbon, Portugal. pp.559-575, 10.1007/978-3-642-23774-4_45. hal-01590584

HAL Id: hal-01590584

https://inria.hal.science/hal-01590584v1

Submitted on 19 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# A Taxonomy of Microinteractions: Defining Microgestures based on Ergonomic and Scenario-dependent Requirements

Katrin Wolf¹, Anja Naumann¹, Michael Rohs², Jörg Müller¹

¹ Deutsche Telekom Laboratories, TU Berlin, Ernst-Reuter-Platz 7,
10587 Berlin, Germany

² LMU Munich, Amalienstr. 17, 80333 Munich, Germany
katrin.wolf@acm.org, anja.naumann, joerg.mueller03@telekom.de,
michael.rohs@ifi.lmu.de

**Abstract.** This paper explores how microgestures can allow us to execute a secondary task, for example controlling mobile applications, without interrupting the manual primary task, for instance, driving a car. In order to design microgestures iteratively, we interviewed sports- and physiotherapists while asking them to use task related props, such as a steering wheel, a cash card, and a pen for simulating driving a car, an ATM scenario, and a drawing task. The primary objective here is to define microgestures that are easily performable without interrupting or interfering the primary task. Using expert interviews, we developed a taxonomy that classifies these gestures according to their task context. We also assessed the ergonomic and attentional attributes that influence the feasibility and task suitability of microinteractions, and evaluated their level of resources required. Accordingly, we defined 21 microgestures that allow performing microinteractions within a manual, dual task context. Our taxonomy poses a basis for designing microinteraction techniques.

**Keywords:** gestures, microinteractions, dual-task, multitask, interruption.

## 1 Introduction

Human-computer interactions are to a great extent defined by hardware design such as the size limitations and the interconnections of the hardware components. For instance, the size of current smart phones is mainly determined by the screen size necessary for watching multimedia content or browsing the internet.

Novel concepts of interaction design and HCI research tend to split the interface into specialized components, especially for separating the hardware that processes the user input [5, 8, and 15]. For example, Loclair [8] uses a depth camera for tracking pinch gestures; Harrison [5] measures body transmitted acoustic signals that are generated by tapping a finger against other fingers or the forearm; and Saponas [15] is using EMG to recognize finger pressure and finger taps. These works focus on the input and sensing techniques for tracking hand gestures for microinteractions.
Microinteractions, defined by Ashbrook as short-time interruptions of primary tasks [1], can have huge benefits in allowing mobile application control in parallel to ongoing primary tasks and could significantly expand the set of tasks we could perform on-the-go. Chewar [2] defined secondary tasks as those which can take place concurrently with the primary task. However, there is a research gap in investigating microinteractions from the task-driven perspective and from the human point of view [17].

We understand microinteractions as interactions that are task-driven and goal oriented, and which may include system feedback. They can be evaluated with traditional usability metrics such as effectiveness, efficiency and user satisfaction. In contrast, microgestures are actual physical movements, e.g. of fingers, which are recognised by the system, and where the system reacts upon. Microgestures are part of microinteractions. Within the related work of microinteractions, the main focus is on short-time manual motor interruptions, or on manual synchronous tasks. We investigate microinteractions that can be performed synchronously. The attentional resources then have to be used alternately or in parallel.

This paper explores and identifies microgestures and finger movements that are performable and does not draw significant attention away from the primary manual task which is to be done in parallel. In deciding the manual primary task, we focused on manual grasp research that is done in the rehabilitation and medical science areas.

Feix [4] developed a grasp taxonomy that compared 14 grasp taxonomies based on 92 years of human hand's research. He identified 33 different human natural grasps and classified them into 3 main types: palm, pad, and side. We abstracted this taxonomy and related it to our research interest: microgestures performed alongside manual tasks (see Table 1). The left three columns of the table shows the original main grasp types of Feix' taxonomy and describes one specific example for each type. The right column shows which free movement potentials we identified for the taxonomy's main grasp types. For investigating microinteractions that are meant to be executable alongside manual tasks, we have chosen 3 exemplary tasks: each one is using one grasp of one main group of Feix' taxonomy. Thus, we aim for ensuring research results that are scalable to a wide range of manual activities.

Primary tasks, such as driving a car or holding objects, do not need our complete cognitive attention nor are all fingers strictly involved in these processes. This allows for performing a second task at the same time. This task can be related to a different context like answering the phone while driving a car. Alternatively, controlling mobile applications by microinteractions could also offer the opportunity to apply subtasks through adding augmented function to the primary task without interruption. For instance, the input for many mobile applications in the automotive context, such as setting up the navigation system, controlling the music player, or opening and shutting the car windows, could be realized by microinteractions that are performable without releasing the steering wheel and therefore not interrupt the manual effort of the primary task.
<table><caption>**Table 1.** Microgesture options during ongoing manual tasks: Analysis of Feix's grasp types: Palm, Pad, and Side, into which all human grasps can be categorized. Fingers are counted starting from the thumb.</caption><thead><tr><th>Grasp type<br>(Feix [4])</th><th>Description<br>(Feix [4])</th><th>Involved hand-parts<br>(Feix [4])</th><th>Potentially<br>still movable<br>hand-parts</th></tr></thead><tbody><tr><td><strong>PALM</strong><br>(e.g. Steering a car)</td><td>Medium wrap</td><td>Low power grasp performed by 2-directional force between palm (finger 2-5) and abducted thumb</td><td>Particular fingers and thumb</td></tr><tr><td><strong>PAD</strong><br>(e.g. Inserting a cash card into an ATM)</td><td>Precision grasp</td><td>2-directional force between abducted thumb and index finger</td><td>Finger 3-5: middle, ring, and little finger</td></tr><tr><td><strong>SIDE</strong><br>(e.g. Drawing with a stylus on a graphic tablet)</td><td>Dynamic tripod</td><td>2-directional force between:<br>a) added thumb and middle finger while index finger stabilizes or<br>b) thumb and index finger while middle finger stabilization</td><td>Ring, little finger<br>Stabilizer:<br>index finger or middle finger</td></tr></tbody></table>

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

For the palm grasp for example, we have chosen driving a car as primary task that allows microgesture commands such as tipping or dragging at the steering wheel (see Fig.1).

<figure><img src="image_5.png" /><figcaption>**Fig. 1.** Performable microgestures while steering a car with a palm grasp: Tipping fingers on the wheel or dragging it with the thumb.</figcaption></figure>

In contrast to the chance of enriching the primary task conceptually by allowing a secondary task to be performed simultaneously; there is a risk that the performance of
the primary as well as of the secondary task might decreases, because of attentional
deficit. [3, 20]

## 2 Related Work

We relate our work to research that is investigating microinteractions performed by hand gestures. We focus on the effect of multitasking on motor and attentional efforts, on gesture-based interaction techniques as well as on wearable gesture tracking systems that do not limit the hand skills like data gloves do by reducing the touch sensitivity of the hand.

Within the human-factors related research, multitasking is investigated focusing on task interruptions and attentional issues of both: the primary and the secondary task. While Wexelblat [19] and Quek [13] claim that gestures are not natural for computer interactions because they only represent a small part of human communication, Karam [7] suggests that this “small part” is potentially well matched to secondary interactions. McCrickard [9] investigated the effects of distraction and recovery caused to a primary task (editing a text document) by a secondary task interruption, which was a notification for receiving an instant message. For the specific case of dual-task-microinteractions, there is a gap of research about how to design dual-task scenarios and how to select microgestures. For keeping the performance stable, there are two strategies: alternating two tasks or performing them in parallel. This is possible if at least one task can be performed with a certain level of automation and therefore requiring limited attention.

Wickens’ Multiple Resource Theory (MRT) Model describes that two actions can be done in parallel if at least one has reached an automated level through learning [20]. Based on Wickens’ Multiple Resource Theory [20], Oulasvirta [12] developed the Recourse Competition Framework (RCF). He is investigating cognitive resources when users are on the move. Oulasvirta explains that the resources are partly reserved for passively monitoring and reacting to contexts and events, and partly for actively constructing them. This model suggests that the resources for competitive task interactions alternate through breaking down the primary fluent interaction for up to four seconds.

Another research field that concerns about microinteractions and their trackability and classification is computer science. Computer vision based gesture tracking for identifying pinch gestures has been investigated by Loclair [8]. Vardy [18] tracks finger flexion with a camera integrated in a wrist band. Howard [6] uses optical detectors (that are also integrated in a wrist band) for measuring LED light that is reflected by the fingers. Harrison [5], Saponas [15], and Rekimoto [14] measure hand gestures using body transmitted signals, such as acoustic signals, EMG, and electrodes that display forearm movements by capacitive sensing.

So far, several multitasking scenarios and interaction techniques have been explored and tracking technologies for microinteractions have been developed and evaluated. But there is still a research gap in classifying microinteractions regarding their ergonomic dual-task potential. We investigate which microgestures might be best suited when applying secondary tasks in addition to certain exemplary primary
tasks. Therefore we aim to develop a taxonomy based on fundamental ergonomic and anatomic hand research [4]. Our taxonomy can serve as a basis for developing novel microinteraction-based interfaces.

## 3 Method

The goal of our study was to generate taxonomy for microinteractions by listing and evaluating all microgestures that are performable alongside the main grasp types. The taxonomy aims to develop a general hand gesture set as well as to display ergonomic issues related to hand gesture performance, the necessary attention to perform the gestures, and the risk that the gesture is performed unintentionally as a natural movement and therefore would be misinterpreted as an input command.

A common method for defining gestures in the HCI field is to involve users in the design process [21]. To create a gesture set that already contains gestures of good feasibility and to generate valid data about how the majority of the users will be able to perform these gestures while continuing a manual task, we decided to involve experts, who know about the motor abilities and limitations of the majority of the users. Therefore we interviewed one sports therapist and three physiotherapists separately and asked them to evaluate a gesture set using props (see Fig. 2) regarding ergonomic and scenario-related aspects as well as to find more gestures that might suit the use case.

<figure><img src="image_6.png" /><figcaption>**Fig. 2.** The participants are testing the feasibility of hand gestures while (1) holding a steering wheel (2) targeting a cash card, and (3) drawing with a pen.</figcaption></figure>

We interviewed the experts separately in two sessions (see Fig. 3). We started the first session with a prepared set of 11 hand gestures, which were graphically presented to be evaluated by the experts. This initial gesture set consisted of seven palm-gestures, two pad-gestures, and two side-gestures, which were already used within microinteraction research projects [1, 5, 6, 8, 14, 15, 18]. For each gesture, we asked the experts to evaluating its performance ability by answering the following questions:
*Feasibility.* How easy is the hand gesture performable regarding ergonomic aspects when it is done eyes-free?

*Limitations.* Which ergonomic aspects limit the hand gesture performance?

*Attention.* Does the pure gesture performance require low, medium, or high attention?

*Risk of confusion with natural movements.* Could the gesture be randomly performed during the task?

<figure><img src="image_7.png" /><figcaption>**Fig. 3.** Experiment walk through: We interviewed the experts separately in two sessions. The first started with a pre-defined gesture set, which the experts evaluated and completed using props. An open questionnaire completed the first session. Within the second session, the new collected gestures were evaluated by the experts by walking through the use cases with help of props again. The result was a completed and evaluated gesture set that shows feasible hand gestures that can be performed while continuing a grasp-based task.</figcaption></figure>

For evaluating the different performance parameters in phase 1-3 and 5-7, we used different scale ranges: For the feasibility, we asked them to distinguish between easy (+) and hard (-). The required attention was valued at “low”, if the gesture execution was easily performable without influencing the main task performance. The value was “medium”, if the gesture execution required some of the attention away from the main task. The value was “high” if executing the gesture needed visual attention or if the main task might be interrupted. Within the evaluation section, we took notes of the
verbal comments. Within the creation section, we took photos and drew sketches of the gestures the experts were performing.

After all evaluations of given gestures in one session, we asked the experts within the creation section to describe and perform further gestures that suit the specific context. We took pictures of these new identified gestures and added them as a graphical presentation to the gesture set for the next interview session.

The first sessions finished with an open interview about the experts' general opinion about the idea to support a manual main task through microinteractions.

## 4 Results

The outcome of our iterative interviews was a list of 21 expert evaluated micro-gestures: 17 palm-, 2 pad-, and 2 side-gestures, as shown in figure 4 and described in greater detail in table 2.

<figure><img src="image_8.png" /><figcaption>**Fig. 4.** The expert-defined and evaluated hand gesture set. The experts found 17 gesture types for the driving scenario (A). The card targeting scenario (B) and the stylus scenario (C) just contain 2 gesture types each. Most gesture types have several sub-types by performing them with different fingers (index, middle, ring, and little finger). Moreover the same gesture results in a different sub-type (e.g. touch, tab, or press), if it is performed with different acceleration or duration (see Table 2).</figcaption></figure>

The very similar evaluation data of the different interviews regarding the valuation of the microgesture's required motor and attentional effort, allowed for comparing and concluding the results into one single table (see Table 2). The opinions we collected during the interviews are subjective expert arguments. In case there were different opinions about the feasibility or attentional efforts of a microgesture, we chose the more negative ones in order to exclude the less feasible gestures from further examination, and to make sure that the taxonomy will work for a large number of users. In the following, the results in Table 2 will be described in detail.

Arguments for valuing ergonomic issues were classified in sub-clusters: feasibility arguments that described why some gestures were hard or impossible to perform (limitations because of the shape of the grasped object or because of the anatomy of human hand). We identified arguments which described how well the primary and the secondary task fit together into a situation with simultaneously performed tasks. Within this category, we asked in particular for two aspects: attention and risk of confusion. The attention concerning comments describe if the in parallel performance
of certain gestures requires high or low attention. The risk of confusion comments value the risk that a gesture is performed randomly as natural gesture or movement.

## 4.1 Feasibility and limitations

We asked the experts to show us feasible hand gestures. In some cases, certain gestures have circumstance-dependant feasibility. For instance, the feasibility of touching, pressing, and tapping the fingers on the thumb while holding a steering wheel is dependant on the finger length and the wheel diameter (see Fig. 5).

<figure><img src="image_9.png" /><figcaption>**Fig. 5.** shows the feasibility of the third gesture of table 1: The thumb can be tapped easily with the middle (2) and the ring finger (3) while holding a steering wheel. But depending on the wheel diameter tapping the thumb with the index (1) or the little finger (4) can be difficult, especially for people with small hands.</figcaption></figure>

There are mainly two classes of limitations in regard to the feasibility of microgestures. On one hand, the limitation is related to the physical objects that are to be grasped, for example, the size of the diameter of a steering wheel. On the other hand, feasibility is also limited by biomechanics, for example, it is difficult to move one finger without slightly moving its neighboring fingers as well.

There was a significant difference in feasibility between the index, middle, ring, and little finger, while performing some hand gestures, such as tapping a single finger on the thumb (Tab. 1, gesture 3). All experts were sure that the majority of the users will be able to perform an index-finger tap without any problems. Also, to move the little finger separately from the others was not a problem at all. The flexibility of the middle finger was a bit worse than of the index finger, but it was still feasible. However, the ring finger is always difficult to stretch separately. The degree of inflexibility varies individually; but the ring finger is considered to be the least feasible.

The sports scientist expert explained this motor limitation and defined anatomic reasons like the connection between our muscles, sinews, and the fingers (see Fig. 6). Humans have more than 40 muscles to move the arm, hand, and fingers. If we aim to stretch the ring finger out from a palm grasp; two muscles (M. flexor digitorum profundus & M. flexor digitorum superficialis) are bending synergistically the index, middle, and little finger to bring them into the palm position. In addition another muscle is responsible for stretching the ring finger (M. extensor digitorum) but because this muscle is also responsible for stretching the other fingers and because the ring finger has a physical connection to the middle finger (Connexus intertendineus), the middle finger will always move a bit in the same direction as the ring finger does.
The little and the index finger are more independently movable because they have
their own muscles for stretching.

<figure><img src="image_10.png" /></figure>

**Fig.6.** shows the anatomic connection between the fingers that is responsible for the separation problem of the ring finger. Fig. 6 is a simplification of a figure in Spalteholz' Anatomy of Human [16].

This means that in designing microgestures, it is preferable to focus on the index finger. In the case that a microgesture involves the ring finger, we will need to design it bearing in mind that the little and the middle finger will move slightly as well (see Fig. 7).

<figure><img src="image_11.png" /></figure>

**Fig. 7.** shows the difficulty of stretching the ring (2) and middle (3) finger separately. Stretching the little (1) or the index (4) finger is much easier because of human hand's anatomic architecture which is shown in greater detail in Fig. 6.

**4.2 Risk of confusion**

If commands are released by body movements, there is a risk that subconsciously executed natural movements can be misinterpreted as commands. For example, tabbing the steering wheel while driving a car (tab. 1, gesture 16) is a common behavior while waiting at the crossroads or listening to music. Reaming the thumb
against the index finger would be expected while cooking, eating or putting salt on food, but while driving the risk of a ream-gesture occuring as a randomly executed natural movement is expected to be low.

4.3 General idea

Besides gesture evaluation, we also did a questionnaire and collected the verbal comments of the experts on the general idea of allowing a secondary task alongside a continuous primary one. The opinion about the benefit of performing two tasks in parallel was different from one scenario to another. All experts think there is a huge benefit in being able to control a secondary task while driving a car. An example that is often used to support this argument is that drivers are anyway performing secondary tasks such as setting up the navigation system, controlling automotive functions, or using mobile devices like cell phones while steering a car. The concept of controlling these devices or applications without releasing the steering wheel was valued positively for security arguments. The scenario of performing hand gestures while inserting a cash card into an ATM was not liked at all. None of the experts thought in parallel tasks could have a benefit for this use case. The last scenario about pen computing (e.g. drawing with a pen- or stylus-like input device on a graphic tablet) was modified during the interviews. Three of the experts thought that the possibility to change the stroke width or the color while drawing would have a bad effect on the precisions of the primary task but all of them said that having these options during short time interruptions could benefit the primary task. The flow of drawing would not be interrupted and therefore the task could be designed to be more comfortable than if a color selection would have to be done by keyboard or button-selection.

In general, the experts think palm grasp tasks best suit dual-task scenarios because these tasks are often low precision tasks and therefore require lower attention than pad or side grasp tasks.
<table><caption>**Table 2.** Microinteraction taxonomy. I=Index Finger, M=Middle Finger, R=Ring Finger, L=Little Finger, Th=Thumb, +=easy, - = difficult.</caption><thead><tr><th>Gesture</th><th>Action</th><th>Ergonomic</th><th>Scenario compatibility</th></tr></thead><tbody><tr><td colspan="4"><strong>Palm-grasp gestures</strong></td></tr><tr><td rowspan="3">(1)</td><td>(a) Tab</td><td><strong>Feasibility</strong><br>Index (I):easy+<br>Middle (M): +<br>Ring (R): +<br>Little (I): diff. -</td><td><strong>Attention</strong><br>Low:<br>Th (thumb), I,<br>M, L<br>High: R, M</td></tr><tr><td>(b) Touch</td><td><strong>Limitation</strong><br>By relation of<br>finger length<br>and hold object<br>diameter, i.e.<br>steering wheel</td><td><strong>Risk of confusion</strong><br>Risk to be a<br>randomly<br>performed<br>natural move:<br>high</td></tr><tr><td>(c) Press</td><td></td><td><strong>Attention</strong><br>Higher than<br>Touch-gesture,<br>pressure rate is<br>hard to control<br><strong>Risk of confusion</strong><br>High</td></tr><tr><td rowspan="3">(2)</td><td rowspan="3">(a) Tab<br>(b) Touch<br>(c) Press</td><td><strong>Feasibility</strong><br>I: +, M: +, R: +,<br>L: -</td><td><strong>Attention</strong><br>Higher than (1);<br>Hard to<br>distinguish from<br>(3)</td></tr><tr><td><strong>Separation</strong><br>-: M+R</td><td><strong>Risk of confusion</strong><br>High</td></tr><tr><td><strong>Limitation</strong><br>By holding<br>object diameter</td><td></td></tr><tr><td rowspan="3">(3)</td><td>(a) Tab</td><td><strong>Feasibility</strong><br>M, R: +<br>I, L: -</td><td><strong>Attention</strong><br>Higher than (1);<br>Hard to<br>distinguish from<br>(2)</td></tr><tr><td>(b) Touch</td><td><strong>Separation</strong><br>-: M+R</td><td><strong>Risk of confusion</strong><br>High</td></tr><tr><td>(c) Pinch</td><td><strong>Limitation</strong><br>Object diameter</td><td></td></tr><tr><td rowspan="3">(4)</td><td rowspan="3">Flip</td><td><strong>Feasibility</strong><br>I: +, M: +, R: +,<br>L: -</td><td><strong>Attention</strong><br>Low</td></tr><tr><td><strong>Separation</strong><br>No problem</td><td><strong>Risk of confusion</strong><br>Low</td></tr><tr><td><strong>Limitation</strong><br>Object diameter</td><td></td></tr><tr><td>(5)</td><td>Drag&Drop index on thumb</td><td><strong>Feasibility</strong><br>Just partly<br>possible<br>because of<br>object diameter</td><td><strong>Attention</strong><br>Medium<br><strong>Risk of confusion</strong><br>Medium</td></tr></tbody></table>
<table><thead><tr><th>Gesture</th><th>Action</th><th>Ergonomic</th><th>Scenario compatibility</th></tr></thead><tbody><tr><td colspan="4">Palm-grasp gestures</td></tr><tr><td>(6)</td><td>Ream</td><td><strong>Feasibility</strong><br>I: +, M: +, R: +,<br>L: -<br><strong>Limitation</strong><br>hold object<br>diameter (L)</td><td><strong>Attention</strong><br>Low<br><strong>Risk of confusion</strong><br>Low</td></tr><tr><td>(7)</td><td>Circle sidewise</td><td><strong>Feasibility</strong><br>I: +, M: +, R: +,<br>L: -<br><strong>Separation</strong><br>No problem<br><strong>Limitation</strong><br>Object diameter</td><td><strong>Attention</strong><br>Individually<br>different (+, -)<br><strong>Risk of confusion</strong><br>Low</td></tr><tr><td>(8)</td><td>Drag fingers<br>around the wheel</td><td><strong>Feasibility</strong><br>-</td><td><strong>Attention</strong><br>Medium<br><strong>Risk of confusion</strong><br>Medium</td></tr><tr><td>(9)</td><td>Drag&amp;Drop<br>middle on index</td><td><strong>Feasibility</strong><br>To complicated</td><td><strong>Attention</strong><br>High</td></tr><tr><td>(10)</td><td>Snip</td><td><strong>Feasibility</strong><br>+</td><td><strong>Attention</strong><br>Low<br><strong>Risk of confusion</strong><br>Low</td></tr><tr><td>(11)</td><td>Tap the wheel</td><td><strong>Feasibility</strong><br>I-L: +, Th: -</td><td><strong>Attention</strong><br>Low<br><strong>Risk of confusion</strong><br>High</td></tr><tr><td>(12)</td><td>Thumb up</td><td><strong>Feasibility</strong><br>+</td><td><strong>Attention</strong><br>Low<br><strong>Risk of confusion</strong><br>Low</td></tr><tr><td>(13)</td><td>Drag&amp;Drop<br>thumb on finger<br>nails</td><td><strong>Feasibility</strong><br>Over I, M, R: +<br>L: -<br><strong>Limitation</strong><br>Object diameter</td><td><strong>Attention</strong><br>Low<br><strong>Risk of confusion</strong><br>Low</td></tr></tbody></table>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>
<table><thead><tr><th>Gesture</th><th>Action</th><th>Ergonomic</th><th>Scenario compatibility</th></tr></thead><tbody><tr><td colspan="4"><strong>Palm-grasp gestures</strong></td></tr><tr><td>(14)</td><td>Drag&amp;Drop thumb on index-side</td><td><strong>Feasibility</strong><br>Just partly possible because of object diameter<br><strong>Limitation</strong><br>Object diameter</td><td><strong>Attention</strong><br>High<br><strong>Risk of confusion</strong><br>Medium</td></tr><tr><td>(15)</td><td>Circle clockwise & contra-clockwise (CW & CCW)</td><td><strong>Feasibility</strong><br>I, M: +, R, I: -<br><strong>Limitation</strong><br>Object diameter</td><td><strong>Attention</strong><br>Individually different (+, -), but high for CW-/CCW-distinguishing<br><strong>Risk of confusion</strong><br>Low</td></tr><tr><td>(16)</td><td>Drag thumb along object</td><td><strong>Feasibility</strong><br>+</td><td><strong>Attention</strong><br>Low<br><strong>Risk of confusion</strong><br>Low</td></tr><tr><td>(17)</td><td>Drag thumb around object</td><td><strong>Feasibility</strong><br>-</td><td><strong>Attention</strong><br>Medium<br><strong>Risk of confusion</strong><br>Low</td></tr><tr><td colspan="4"><strong>Pad-grasp gestures</strong></td></tr><tr><td>(18)</td><td>Tab</td><td><strong>Feasibility</strong><br>I, R: -<br>M, I, M&amp;I: +</td><td><strong>Attention</strong><br>High</td></tr><tr><td>(19)</td><td>Drag middle finger above object</td><td><strong>Feasibility</strong><br>+</td><td><strong>Attention</strong><br>High</td></tr><tr><td colspan="4"><strong>Side-grasp gestures</strong></td></tr><tr><td>(20)</td><td>Tab I or M on object</td><td><strong>Feasibility</strong><br>I.<br>While drawing:<br>-<br>II.<br>While holding:<br>+</td><td><strong>Attention</strong><br>I: High<br>II. Low<br><strong>Risk of confusion</strong><br>I: High<br>II. Low</td></tr><tr><td>(21)</td><td>Drag Index or Middle finger on stylus up / down</td><td><strong>Feasibility</strong><br>While drawing:<br>-<br>While holding:<br>+</td><td><strong>Attention</strong><br>I: Low, M: High<br><strong>Risk of confusion</strong><br>Low</td></tr></tbody></table>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>
# 5 Discussion

The microinteraction taxonomy shows that the design of microgestures, as well as their evaluation concerning usability related issues (e.g. ergonomic issues and scenario compatibility), is extremely dependent on the use context. This defines the primary task and rules the choice of the grasp type that is used to solve this task. The static gesture design as well as its feasibility (see table 1, column 1 & 3), is mainly influenced by grasp-related options such as hand anatomic limitations, but also ergonomic issues that are defined by objects and the character of grasping of the primary task. For instance, as explained below in greater detail, a low power palm grasp that is mostly used while driving, allows a lot of microgestures because releasing a finger from the steering wheel does not interrupt the task. Moreover, the primary task determines the attentional resources that are available to perform secondary task commands realized by microgestures.

## 5.1 Palm-grasp gestures

A low power palm grasp gesture allows for a great number of simultaneously performed microgestures without releasing the grasp. Palm related primary tasks that have a long duration require little attention by becoming an automatically performed process and leave a large part of the hand resources quite uninvolved. Thus, low power grasps seem well suited to be augmented by a large variety of microinteractions. Depending on the character of the primary task, some microinteractions have a high risk of being performed unintentionally during the primary task. Tapping on the steering wheel could done while listening to music and drumming fingers on the wheel. To differentiate natural movements from input commands, three opportunities are possible for generating a gesture set:

1. Using a push-to-gesture event for telling the system that the parallel or subsequent movement is an intentionally performed command.

2. Designing commands as a combination of two gestures for reducing the chance of performing this couple unintentionally.

3. Defining design styles, e.g. rhythmic pattern, based on movements which are usually not done naturally in the primary-task-related context.

## 5.2 Pad-grasp gestures

Pad-grasp primary tasks such as inserting a cash card into an ATM machine due short/card slot?, use the 2 directional finger-thumb-force permanently, and require a high level of precision and short-term concentration. This was shown by our expert through demonstrating the failed attempt to perform both tasks in parallel. An added microinteraction would require interrupting or slowing down the primary task for a short time while performing the microgesture. According to the expert opinion, the interruption of the precisely short-term primary task is not acceptable because performing hand movements quickly and accurately does not allow microinteractions in parallel. Any finger movements would disturb targeting the cash card into an ATM
by dismissing the target or extending the targeting time. Targeting and performing microgestures at the same time without risking high error rates on one or even both tasks is not possible. Moreover, the available hand resources for performing microgestures while interrupting the pad-grasp but still holding the tool are very limited.

## 5.3 Side-grasp gestures

Performing microgestures alongside a side-grasp drawing is hardly possible. Drawing is a highly precise manual task which is built on accurate hand movements and does not allow for the moving of fingers at the same time without having a negative effect on the quality of drawing. However, brief interruptions (to stop drawing but to continue to hold the stylus) would allow for microinteractions. There are just a few possible microgestures while holding a stylus but these are quite easy to perform and require low recognition effort.

## 5.4 Dual-task suitability

In summary, several parameters have an effect on how well two tasks suit a dual-task scenario, such as the duration of both tasks and the attention (alternate versus in parallel effort) that is necessary to solve the tasks without increasing the efficiency or *decreasing the?//effectiveness of the task*. The suitability of two tasks depends on the level of required precision and the required attention as well as on the synchrony of these requirements.

Comparing the evaluated tasks, we argue that primary tasks, which have a long duration, are performed automatically and require low attention and motor effort, are suitable for simultaneous microinteractions. Of the conditions we evaluated, the palm grasp is the most promising for leaving enough motor resources for simultaneous hand gestures.

# 6 Conclusion and Design Guideline

Gestural interfaces lack the affordances and constraints that are readily provided by other interfaces, such as graphical and tangible ones [10, 11]. In particular, it is difficult to inform users what they are able to do, what they are currently doing or what they have just done. Because of this, gestural interfaces and in particular microgestures are not to be understood as a replacement for other kinds of interfaces, but rather as enabling novel ways of interaction. There are still many open questions to be answered, especially regarding the interaction opportunities and feedback representation.

Our taxonomy mainly investigated ergonomic interaction opportunities of microinteractions and can be used as a basis for designing microinteraction techniques for manual dual-task scenarios: First, the scenario has to be analyzed for defining the limits and requirements for microgestures. A gesture set can then be defined by
looking at the formal structure of the chosen gestures. Lastly, a gesture driven
decision about the sensory and tracking requirements of the hardware can be made.

**6.1 Dual-tasking design**

For a formal scenario design, we proposed two synergetic strategies: the economics of attentional and motor budgeting.

The selection of the primary and the secondary task is reasoned by the usage of different attentional resources. Our primary and secondary tasks used equal modalities by requiring tactile bio-feedback and kinesthetic self-awareness. An automatically performed primary task requires low attention [20]. This allows paying attention for simultaneously secondary tasks performance, such as microinteractions. These circumstances allow the economics of two tasks performances in parallel (Tab. 2, column 4). The example of steering a car, if it is done by people with some practice, represents an automatically performed task with low attention. Controlling the navigation system by microgestures could be a secondary one that requires attention.

The primary task defines the usage of motor resources as well as free potentials and available hand motor skills that can be used for simultaneous tasks. The grasp type that is performing the primary task (palm, pad, side) defines the motor resources which are used in the primary task (see Table 1, column 3). Our taxonomy identifies microinteractions executable in parallel based on free motor resources (Table 2, column 1-3) and allows the creation of microgesture set for commanding the secondary task.

**6.2 Interface design**

The developed gesture set defines requirements necessary for the interface design and the gesture tracking technique of microinteractions. For example tap-interactions should be tracked by a technology that provides a sequence of movement data like accelerometer. Gestures that are based on finger pressure are defined by vectored force applied on an object's or on skin surface and could be tracked by sensors that measures muscle activities such as EMG. The different tracking technologies shall be discussed for their data quality, and their interaction usability under different conditions given by both the microgesture design and the primary tasks.

There are some primary task-driven requirements for the sensor selections beside selecting the best suited sensors to measure formal gesture parameters. Covering the finger tips with interface components such as touch sensors would limit the tactile feedback (sense of touch) of the finger and the ability to conduct highly precise tasks. Moreover, the size and placement of the hardware could affect both the primary task and the ability to perform the input gestures. The interface design should not be cumbersome to wear and should be as small and unobtrusive as possible.
# 7 Further Research

The developed taxonomy serves as an analytic basis for systematic microinteraction design. As a next step, we intend to ask users to perform these microinteractions while performing a primary task and ask them to rate the feasibility of the gesture as well as scenario-related usability.

So far, we increased the hand gestures regarding their ergonomic structure and did not analyze their semiotic potentials. But within our interviews, we also received suggestions on what the gestures could communicate. For instance, the thumb-up-gesture (see Table 2, gesture 13) was commented to suit for okay-commands like answering the phone or selecting a pointed menu item. The taxonomy contains some more meaningful gestures, such as forming the index finger and the thumb to an “O” for communicating an “Okay”. A snip gesture (see Table 2, gesture 12) could mean cutting something, and to put the thumb up (see Table 2, gesture 13) is also commonly understood as “Okay”. When the gestures are linked to specific meanings and commands, it will be necessary to not just pay attention to the feasibility of a gesture but also to its potentials of association, guessability, and meaning.

# References

1. Ashbrook, D. Enabling Mobile Microinteractions, Doctoral Theses, Georgia Institute of Technology 2010

2. Chewar, C.M., McCrickard, D.S., Ndiwalana, A., North, C., Pryor, J., and Tessendorf, D., Secondary task display attributes: optimizing visualizations for cognitive task suitability and interference avoidance, In Proc. Data Visualisation 2002, pp. 165-171

3. Czerwinski, M., Horvitz, E. and Wilhite, S., A Diary, Study of Task Switching and Interruptions, In Proc. Conference on Human Factors in Computing Systems 2004, pp. 175-182

4. Feix, T. et al. Grasp Taxonomy Comparison Sheet, DOI = http://web.student.tuwien.ac.at/~e0227312/documents/taxonomy_comparison.pdf

5. Harrison, C. et al. Skinput: Appropriating the Body as an Input Surface, In Proc. CHI 2010

6. Howard, B, Howard, S. Lightglove: Wrist-Worn Virtual Typing and Pointing, In Proc. ISWC 2001

7. Karam, Maria, A Study on the Use of Semaphoric Gestures to Support Secondary Task Interactions, In Proc. UIST 2003

8. Loclair, C., Gustafson, S., Baudisch, P. PinchWatch: A Wearable Device for One-Handed Microinteractions, In Proc. MobileHCI 2010
9. McCrickard, D.S., Chewar, C.M., Somervell, J.P. and Ndiwalana, A. A model for notification systems evaluation—assessing user goals for multitasking activity. ACM Transactions on Computer-Human Interaction (TOCHI), 10 (4), pp. 312-228

10. Norman, D. A. Natural user interfaces are not natural, interactions, 17, No. 3 (May - June) 2010

11. Norman, D. A., Gestural Interfaces. A Step backwards in Usability, interactions, 17, No. 5 (September - October) 2010

12. Oulasvirta, A., Tamminen, S., Roto, V., and Kuorelahti, J., Interaction in 4-Second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI, In. Proc. CHI 2005

13. Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.-F., Kirbas, C., McCullough, K.E., and Ansari, R., Multimodal human discourse: gesture and speech, ACM Transactions on Computer-Human Interaction (TOCHI), 9 (3). 171-193

14. Rekimoto, J. et al. GestureWrist and GesturePad: Unobtrusive Wearable Interaction Devices, In Proc. ISWC 2001, pp. 21-27

15. Saponas, T. et al. Enabling Always-Available Input with Muscle-Computer Interfaces, In Proc. UIST 2009

16. Spalteholz, W., Spanner, R. Handatlas der Anatomie des Menschen – Erster Teil: Bewegungsapparat, Amsterdam 1960, p. 284

17. Tan, D., Morris, D., Saponas, T., S. Interfaces on the Go, In XRDS. Crossroads. The ACM Magazine for Students, DOI = 10.1145/1764848.1764856, p. 30

18. Vardy, A. et al. The WristCam as Input Device, In Proc. ISWC 1999, pp. 199-202

19. Wexelblat, A., Research Challenges in Gestures: Open issues and unsolved problems, In Proc. International Gesture Workshop on Gesture and Sign Language in Human-Computer Interaction 1997, pp. 1-11

20. Wickens, C.D. Processing resources in attention, In R. Parasuraman & D.R. Davies (Eds.), Varieties of attention, Academic Press, New York 1984, pp. 63-102

21. Wolf, K, Dicke, C., Grasset, R. Touching the Void: Gestures for Auditory Interfaces, In Proc. TEI 2010