# Comparing Input Modalities for Peripheral Interaction:
A Case Study on Peripheral Music Control

Doris Hausen, Hendrik Richter, Adalie Hemme, Andreas Butz

► To cite this version:

Doris Hausen, Hendrik Richter, Adalie Hemme, Andreas Butz. Comparing Input Modalities for Peripheral Interaction: A Case Study on Peripheral Music Control. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. pp.162-179, 10.1007/978-3-642-40477-1_10 . hal-01504882

HAL Id: hal-01504882

https://inria.hal.science/hal-01504882

Submitted on 10 Apr 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Comparing Input Modalities for Peripheral Interaction:
A Case Study on Peripheral Music Control

Doris Hausen, Hendrik Richter, Adalie Hemme, Andreas Butz

University of Munich (LMU), Human-Computer-Interaction Group
Amalienstr. 17, 80333 Munich, Germany

{doris.hausen, hendrik.richter, andreas.butz}@ifi.lmu.de,
hemmea@cip.ifi.lmu.de

**Abstract.** In graphical user interfaces, every application usually asks for the user's full attention during interaction with it. Even marginal side activities often force the user to switch windows, which results in attention shifts and increased cognitive load. Peripheral interaction addresses this problem by providing input facilities in the periphery of the user's attention by relying on divided attention and human capabilities such as proprioception and spatial memory. Recent work shows promising results by shifting tasks to the periphery for parallel task execution. Up to now, most of these interfaces rely on tag-based objects, tokens or wearable devices, which need to be grasped and manipulated, e.g., by turning, moving or pressing the device.

To explore this design space further, we implemented three modalities for peripheral interaction with a desktop audio player application – graspable interaction, touch and freehand gestures. In an eight-week in-situ deployment, we compared the three modalities to each other and to media keys (as the state-of-the-art approach). We found that all modalities can be successfully used in the (visual and attentional) periphery and reduce the amount of cognitive load when interacting with an audio player.

With this work we intend to (1) illustrate the variety of possible modalities beyond graspable interfaces, (2) give insights on manual peripheral interaction in general and the respective modalities in particular and (3) elaborate on paper based prototypes for the evaluation of peripheral interaction.

**Keywords:** Peripheral Interaction, Input Modalities, Audio Control

## 1 Introduction

While writing a research paper, a user might repeatedly switch between a text processing program, an online dictionary and related articles. In addition, she might switch to an instant messenger to change her availability status or open her audio player to skip a song. While the first three activities are directly related to her current primary task – writing the paper – the other two are marginal side activities she either has to do or wants to do, but which are in no direct relation to the primary task. Alt-
hough these activities are simple and fast in themselves, current graphical user inter-
faces make the user switch windows or click small icons and thereby impose context
and focus switches. These switches cause a so-called resumption lag, which describes
the time needed to resume work on the primary task [2]. Users also feel more stressed
and cannot focus well if they are frequently interrupted by secondary tasks [4,24].

Peripheral interaction can reduce these interruptions of the users’ primary focus by
making use of peripheral attention. Current projects in this domain primarily use de-
vices, which are grasped and then manipulated (e.g., Fireflies [8], PolyTags [26]).
This is a logical choice as many peripheral interactions in our (non-digital) daily life
are carried out with the hands (e.g., tying shoe laces while talking) [6]. However,
other types of manual input, such as touch or freehand gestures, are hardly explored
up to now for peripheral interaction. Opening the design space for other means of
peripheral interaction makes room for additional application areas benefitting from
less disruptive secondary tasks.

In order to explore this design space, we compared three input modalities (graspable interaction, touch and freehand gestures) (see Figure 1) in an eight-week in-situ deployment with eight participants. As an example use case, we chose the control of an audio player (play/pause, previous/next song and volume), since it is a common but short secondary task when working on a computer. For comparison, we included the commonly adopted media keys built into many keyboards.

We found that participants embraced all peripheral modalities (graspable, touch, freehand) and were able to use them in the periphery of their attention, which broadens the design space for manual peripheral interaction. Further we discuss the use of paper prototypes as an extension of the design process and in addition to the typical in-situ deployments of peripheral interaction. Finally, we highlight implications for manual peripheral interaction in general (e.g., the dominant hand is preferred even for very simple interactions) as well as some distinct characteristics of each modality.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** User interacting with the peripheral modalities (graspable, touch and freehand) to con-
trol his audio player. The actual study was executed in the participants’ homes, but in order to
preserve privacy, these photos were taken in an office.</figcaption></figure>

2 Related Work

Our work builds on existing work on peripheral interaction and previously explored
input mappings for audio player control.
## 2.1 Peripheral Interaction

Peripheral interaction describes the interplay between several tasks, similar to multitasking. However, multitasking research mostly focuses on interruption management, i.e., finding the best possible moment for interrupting with a secondary task [25]. In contrast, peripheral interaction tries to minimize cognitive load and hence the effect of interruptions. The concept is similar to ambient information [28], but uses the periphery not only for perceiving information, but also for active interaction. Bakker et al. give an overview of attention theories relevant for peripheral interaction design [5].

Projects in the scope of peripheral interaction so far mainly use physical artifacts that can be manipulated by grasping them. Edge and Blackwell [13] propose a token-based task management system intended for “imprecise interaction with independently meaningful, digitally-augmented physical tokens”. Olivera et al. [26] built tag-based tangibles to allow changing the status in a social network or controlling the environment with commands such as *power on* and *off*. Their prototypes are intended for brief interactions while the main interaction focus is somewhere else. NoteLet [7], a bracelet for primary school teachers, facilitates taking notes of observations in class. To do so, teachers can take a photo by squeezing the device on their wrist. FireFlies [8] are also designed to aid teachers for example by giving feedback on the fly. The wearable device can be attached to the clothes. By squeezing a bead (each pupil has one corresponding bead) signals are sent to a light object located at each desk. StaTube [17] is a cylindrically shaped tangible, which can be pushed down and rotated to update one’s Skype status. All these graspable prototypes have been evaluated and show that graspable devices are one possibility to support peripheral interaction.

The Appointment Projection [15] is the only prototype in the scope of peripheral interaction, which relies on freehand gestures to acquire information about the next upcoming event in a calendar or to silence an ambient reminder animation. Similarly, related fields such as eyes-free interaction – which is in line with the concept of peripheral interaction, but does not address its full scope (e.g., interaction in the attentional periphery) – made use of gestural input (e.g., [20]). In addition to related work on freehand gestures, there exists work on touch-based input in parallel to the classic desktop setup (e.g., [10,21]), which does not explicitly address the characteristics of peripheral interaction. However, findings from these works encourage us to apply touch and freehand interaction to peripheral interaction.

## 2.2 User Interfaces for Audio Player Control

Playing music with instruments or digital systems or devices (such as, for example the reacTable* [19]) is usually a focused task. In contrast, monitoring and controlling music is an everyday task that is often performed as a secondary task. As we are particularly interested in the latter we focus on audio player control in this section. A large number of prototypical audio player interfaces have been proposed and many of them rely on the input modalities we want to evaluate for peripheral interaction:

*Graspable:* Graspable interfaces for audio player control can mostly be considered cube- or knob-based designs: The Gesture Cube [22] detects the movements of the
user's hand and controls parameters such as volume or play/pause. The Music Cube [1] further presents a button that can be pressed and rotated. Ferscha et al. [14] present a 'key knob', which allows both continuous and discrete control of a music application. Andersen [3] also compares a knob to media keys and found that the knob was preferred over the keys. Butz et al. [11] present additional tangible audio player UIs, some of which are knob- or cube-like or meant to blend into the periphery.

***Touch:*** On mobile touch-based devices, sweeping and tapping the screen can let the user control a music player without looking at the UI [27]. Döring et al. [12] adapted this interaction style for cars on a touch-sensitive steering wheel and reduced visual demand during interaction. Further, several commercial products now use gestural input on touch devices to remotely control a music application on a PC<fnref n="1" />.

***Freehand:*** Strachan et al. [29] control a music player by placing a device near different body parts. This gestural input method utilizes inertial sensing and pattern recognition. Other solutions use sensor-equipped gloves and interpret predefined hand movements [9]. Additionally, the accelerometer in mobile devices is used to track geometric forms drawn in midair, which are mapped to commands [23].

# 3 Designing the Modalities for a Peripheral Music Controller

We selected the specific interaction sets in a two-step process: We analyzed existing systems using graspable interfaces, touch and freehand gestures, extracted interaction mappings and used them in a paper prototype study to identify the most suitable ones.

## 3.1 Interaction Possibilities

There are well-established symbols (▶ II ◄◄ ►►) representing the most common functions of audio equipment and media software. However, they cannot be used here, because peripheral interaction is supposed to work with minimal visual attention and gestural UIs provide no obvious place for graphical symbols. In the absence of such graphical hints, interaction sets have to be easy to use and to remember. Table 1 proposes possible gestures based on the previously mentioned related work or direct metaphors such as the volume knob. Similar to volume knobs and sliders, we want volume control to be continuous (i.e., the volume gesture continuously changes the volume to the desired level). The other commands (pause/play and next/previous) require a discrete gesture to have an effect (e.g., stop the music).

## 3.2 Study with Paper-Based Prototypes

As a next step in the design process, we presented all possible input mappings, listed in Table 1, to participants in a study based on paper prototypes to identify the most accepted ones and thereby ensure easy and natural interaction for all peripheral devices in the long-term deployment.

<footnote n="1">¹ www.iospirit.com/products/remotebuddy/apiaxremote/#gesturedemovideo (accessed: 06.12.12)</footnote>
<table><caption>**Table 1.** Possible input mappings, and the finally selected one (bold and shown in Figure 3). Some gestures (e.g., one/two finger gestures) were presented individually but are summarized here.</caption><thead><tr><th></th><th>Graspable</th><th>Touch</th><th>Freehand</th></tr></thead><tbody><tr><td>=</td><td><ul><li>click</li><li>double click</li><li>long click</li></ul></td><td><ul><li>one/two finger tap</li><li>one/two finger double tap</li><li>one/two finger long tap</li><li>draw square/arrow</li></ul></td><td><ul><li>hold vertical hand in mid-air/thumbs up</li><li>move horizontal hand up/down</li><li>draw square/arrow</li></ul></td></tr><tr><td>▲</td><td><ul><li>tilt knob left/right</li><li>turn knob left/right</li><li>push down & turn knob left/right</li></ul></td><td><ul><li>one/two finger swipe left/right</li><li>tap left/right area on the surface</li></ul></td><td><ul><li>flick left/right</li><li>thumb left/right</li></ul></td></tr><tr><td>▼</td><td><ul><li>turn knob left/right</li><li>push down & turn knob left/right</li><li>tilt knob up/down</li></ul></td><td><ul><li>one/two finger swipe up/down</li><li>circle left/right (cf. iPod click wheel)</li><li>two finger circle left/right (cf. rotation on touch displays)</li><li>tap top/bottom area repetitive/long</li></ul></td><td><ul><li>move horizontal hand up/down</li><li>grasp gesture up/down or left/right (cf. slider)</li><li>circle with hand/finger left/right</li><li>thumbs up/down</li><li>pinch gesture</li></ul></td></tr></tbody></table>

**Paper Prototypes.** To mimic the different modalities, we built paper prototypes out of white cardboard [32] (see Figure 2) resembling existing artifacts (e.g. touch pads). To symbolize a 3D gesture-tracking device we used a box (HxWxD: 6x8x5cm). For the touch sensitive surface, we used a tilted paper surface (HxWxD: 0-2x13x13cm). Looking at existing graspable systems to control an audio player, two shapes are well established in related work: the cube and the knob. We chose a knob mounted on a pedestal for our design (cf. knobs for volume control on stereos). The graspable paper interface was mounted on a bottom panel (20x20cm for stability) with the help of a brass fastener. The knob (Height x Diameter: 5.5x4cm) could be turned, tilted and pressed down.

**Fig. 2.** Paper prototypes mimicking a knob, a gesture tracking device and a touch sensitive surface.

<figure><img src="image_3.png" /></figure>

**Participants.** We recruited 18 participants (six female) ranging in age from 21 to 31 (avg. 25 years). 14 participants had a background in computer science or math, while four participants had a background in economics or languages.

**Procedure.** We used a repeated measures design. The independent variables were modality (graspable, touch, freehand), command (pause/play, next/previous, volume) and gestures. Modality was counterbalanced. We presented 35 gestures to each user. Within one modality, the gestures were presented in a randomized order for each
command. Participants were free to use their preferred hand for the interaction, but the paper prototype was located in the periphery (i.e., not directly in front of them). Further, participants were instructed to minimize gazing at the prototype and imagine working on another primary task simultaneously. Every gesture was first presented and carried out by the instructor and afterwards carried out by the participants themselves. Participants rated each gesture as “bad”, “ok”, or “good”. Additionally, they had to choose their preferred gesture for each modality and command. Afterwards, they were asked to answer a short questionnaire. Each run took about 30 minutes.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Implemented interaction sets for each modality to control an audio player (pause/play, next/previous, volume). The media keys are located on the keyboard below the arrow keys (pause/play; next/previous) and the spacebar (volume).</figcaption></figure>

**Results.** The most preferred and hence implemented gestures are highlighted in bold in Table 1 and shown in Figure 3.

*Graspable:* To start or stop a song, participants preferred a click (89%). Most participants wanted to switch to the next/previous song by tilting the knob left or right (83%). To change the volume participants wanted to turn the graspable knob left and right similar to a volume knob on a stereo (67%).

*Touch:* Using touch to start/stop the player, participants were undecided whether a one-finger tap (44%) or a two-finger tap (11%) felt more suitable. Some also mentioned that they liked both equally (22%). Generally participants were in favor of a tap (77%). We observed similar results for next/previous. A one- as well as a two-
finger swipe left/right was preferred (94%). Similarly participants preferred a one- and two-finger swipe up/down (as on a mixer console) to change the volume (67%). We therefore decided to support both one- and two-finger interactions for all three commands as Wobbrock et al. also reported similar results [33].

*Freehand:* Participants preferred the vertically oriented hand to pause the music (67%). We had originally intended to use a thumbs-up gesture as the counterpart for play, but this gesture was not well received. Participants suggested using the vertical hand also for play, which we implemented as a tap of the vertical hand (cf. Figure 3). To change the song (previous/next), participants favored a flick gesture (78%). Half of the participants preferred an up/down movement of the horizontally oriented hand to change the volume (50%).

We further asked our participants if they could imagine using a separate device for audio control at home. On a 5-point Likert scale (1 = totally disagree; 5 = totally agree), users rated their willingness with a median of 4. They also expected it to make interaction with the audio player easier (median=4) and that they would enjoy it (median=4). Eight participants (44%) preferred freehand gestures, seven (39%) preferred touch and three users (17%) the graspable interface.

# 4 Building the Peripheral Music Controller

The prototype consists of the hardware for the respective input modalities and the software connection to an audio player.

## 4.1 Hardware

Depending on the modality, different hardware was chosen (cf. Figure 3). We decided to use commercial products wherever possible, in order to ensure a reliable function during the in-situ evaluation.

*Graspable:* To implement the graspable interface, we used the Connexion Space-Navigator, which is originally designed as a 3D mouse. (We had also considered the Griffin Technology PowerMate, but ruled it out because it could not implement all the gestures selected in the pre-study.) The hardware offers six degrees of freedom and an SDK. Pushing down the knob controls the play/pause function, next/previous corresponds to a lateral push or tilt, and volume to a rotation of the knob. Only volume uses a continuous manipulation, meaning that every 60ms the volume is decreased or increased as long as the user rotates the device. All three commands included a threshold to avoid accidental executions.

*Touch:* For the implementation of the touch modality, several hardware options were tested. The Apple Magic Trackpad, which we originally had in mind, did not offer raw data access under Windows and no generic driver, which rendered it unusable for our purpose. The same is true for the Logitech Wireless Touchpad and the Bamboo Pen&Touch. Instead, we used the touch-sensitive surface of an Android Nexus One mobile phone. The screen was set to black without any GUI elements and the device never went to sleep. Participants were asked to keep the mobile phone
connected to their computer via USB for power supply. The phone communicated
with the computer through UDP. A threshold was used for swiping up/down (volume)
and left/right (previous/next) to avoid interference with the tap gesture for pause/play.

*Freehand:* Freehand tracking is based on a preproduction prototype<fnref n="2" /> of a capacitive sensing device. We opted for this prototype because it can easily be put on a desk and detects gestures within a range of 10cm, which is perfectly suitable for our use case because this limits the interaction space and reduces the danger of unintended gestures. Using a Kinect, we would not have achieved this easy setup (i.e., small distance to tracking device). The prototype consists of five capacitive sensors located in the four corners and in the middle. X, Y and Z coordinates are provided for the location of the intruding hand’s center in the electrical field. The API already recognized flick gestures (next/previous) as well as tap gestures (pause/play). Volume control was implemented as a movement along the Y-axis (upwards axis). The box (HxWxD: 19x31x3.5cm) with the tracking device offered a tracking area of 7.5x10cm (HxW).

*Media Keys:* We used the Cherry EASYHUB MultiMedia Keyboard, which offers grouped media keys below the regular keys. The volume control is located on the left, below the space bar. On the right, below the arrow keys, are pause/play and next/previous. Media keys are activated without additional keys (e.g. function key).

**4.2 Software:**

We chose Windows as the prototype platform. The software, implemented in C#, connects the different hardware prototypes with the audio player. The GUI, used only for closing the application, offered a status icon in the taskbar, thereby providing feedback that the application is currently running. We selected iTunes as the audio player for implementation reasons (e.g., logging mouse interactions is possible).

# 5 In-Situ Deployment

Peripheral interaction is intended to blend into the periphery of the user, similarly to ambient information systems. Consequently, a long-term in-situ deployment is the preferred way to explore such a system [18]. All reported medians are based on 5-point Likert scales ranging from 1 = I totally disagree to 5 = I totally agree.

**5.1 Participants and Study Environment**

We carried out our exploration with eight participants (two female) ranging in age
from 21 to 32 (avg. 25). All participants used Windows and iTunes before this study.
Six of them were students of computer science and two working (computer scientist
and engineer). Three participants had a computer setup, which provided media keys,
but two of them had to additionally press the function-key to activate the media keys.
Another two participants had previously had media keys and missed them on their

<footnote n="2">² The manufacturer prefers to stay anonymous for strategic reasons.</footnote>
current keyboard. Additionally, four participants had keys for volume control only. All participants were right-handed.

During the study all devices were used at home. Consequently, their primary tasks were very diverse ranging from leisure usage of the computer (e.g., browsing the web, writing emails) to working (e.g., for university). Participants were free to use the computer in any way. Based on their backgrounds, all participants spent much time at their PC, which was helpful to get comprehensive insights of usage with the peripheral modalities alongside a standard desktop computer during the duration of the deployment. Leaving the PC while music was playing of course also was possible.

**Participants' Music Listening Habits.** Seven participants stated that they listen to music on a daily basis for one to four hours. The other participant listens to music three to four days a week. Our participants stated that they listen to music for half of the time (51%) while at their computer. They tend to select a playlist or an album and then listen to it but skip songs they dislike (median=4.5). They rated their interaction frequency with the audio player as medium (median=3). The most used command was pause/play (median=3.5), followed by volume control (median=3) and next/previous (median=2.5). They wish for a faster interaction with the iTunes player (median=4). Further, they are bothered by the focus switch to iTunes (median=4) and consider the distraction of it high (median=4). Apart from the focus switch, interaction with the mouse and the GUI is not considered to be mentally demanding (median=1).

## 5.2 Procedure

The eight-week in-situ deployment contained five semi-structured interviews with each participant. Every participant tested each modality for two weeks (repeated measures design). The distribution of devices was counterbalanced, i.e., we used a Latin square design to minimize learning effects.

*1st Meeting.* During the 1st meeting, we collected data about the participants' listening habits, usual interaction with their audio player and demographics. We installed the application, set it to auto start and asked them to leave it open whenever their computer was turned on. They were handed their first device and received an instruction about the gestures. At this point, participants did not know which other devices they would be testing in the upcoming weeks.

*2nd to 4th Meeting.* During the 2nd to 4th meeting, which took place in two-week intervals, participants were asked about their experiences with the device they had at home. We asked about the general usage as well as their perceived mental load and the ability to carry out interaction in the periphery. They handed back the old device and received the next one together with the respective instructions.

*5th Meeting.* During the last meeting, participants were interviewed about the last device. Additionally, we carried out a comparative interview about all devices. In the end, the application was uninstalled and the last device was given back to us.

*Logging.* For all modalities, we logged: (1) when and which command (pause/play, next/previous, volume) was executed, (2) which input modality (peripheral device,
mouse, media keys) was used, and (3) whether iTunes was in focus while carrying out a command. Furthermore, we logged (4) the duration that iTunes was opened and music was played.

*Conditions and Variables.* All participants tested all four conditions: Peripheral devices (graspable, touch and freehand) and media keys. Based on our logging data, our dependent variables are number of commands, type of command, input modality used, focus of iTunes, duration of iTunes opened and duration of music listening.

# 6 Results of the In-Situ Deployment

During the deployment, we collected quantitative and qualitative data. A listening session is defined as listening to music without a break longer than 30 minutes [34].

## 6.1 Duration of each Modality

When conducting a long-term evaluation over eight weeks, unexpected events invari-
ably occur. One participant went on a spontaneous holiday. We therefore excluded the
corresponding data from the quantitative evaluation. Another participant unexpectedly
changed the workplace and only spent the weekends at home for the last four weeks
of the study. Furthermore, we cannot control how often participants actually listen to
music. Each modality was tested on average for the following number of days by each
participant: graspable: 14.4, touch: 13.1, freehand: 12.8, and media keys: 15.1.

We logged 391 listening sessions overall. These added up to 12 days, 5 hours, 32 minutes and 2 seconds. An average session lasted 45:03 minutes. The longest listening session we logged lasted 6:48:40 hours. Our participants listened to 5652 songs (average 14.5 songs per session) and they executed 6119 commands (peripheral devices, media keys and mouse).

## 6.2 Frequency of Use

Looking at the probability of a command
being issued during one minute for each
respective modality, the highest probabil-
ity is observed for freehand gestures
(m=93%) (probably some of these logged
interactions are due to tracking issues of
freehand interaction, which is still not as
reliable as other input modalities), fol-
lowed by touch gestures (m=72%) and
graspable interaction (m=51%) (see Fig-
ure 4). Media keys have the lowest prob-
ability (m=20%). Overall we did not ob-
serve any significant change between
each of the two weeks the participants

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Probability of issuing a command for all available input modalities (peripheral device, mouse and if available media keys) during one minute.</figcaption></figure>
used one device. In other words, usages slightly differed between the two weeks for each participant but sometimes they used the peripheral device more, sometimes they used it less during the second week. We assume this is due to external circumstances occurring during an in-situ deployment but not related to the peripheral devices.

## 6.3 Modalities vs. Mouse

Participants were given a peripheral device but of course they were free to use the traditional GUI with a mouse or media keys, if their standard keyboard allowed them to. Still, participants used the additional devices (with the exception of the media keys) significantly more than the traditional GUI (t-tests: graspable vs. mouse *p* = 0.003; touch vs. mouse *p* = 0.003; freehand vs. mouse *p* = 0.006).

## 6.4 Usage in the Periphery

All devices were intended to be used in the (visual and attentional) periphery. Participants quickly familiarized themselves with the modalities (graspable: median=5; touch: median=5; freehand: median=4.5; media keys: median=4.5). However, they mentioned that the location of the media keys as well as the freehand tracking was not immediately clear. Interaction sets were all rated as being intuitive (graspable: median=5; touch: median=5; freehand: median=4.5), but media keys were given a medium rating (median=3), because they required a targeted key press, even though the keyboard layout already emphasized the media keys by their position. Interaction only caused minimal mental load for all peripheral devices (graspable/freehand/touch: median=5), but participants experience more mental load for media keys (median=2.5). When asked whether disruption from the primary task, while interacting with the devices, was low, the graspable device was rated best (median=4.5), while the other modalities were rated nearly indifferent (touch: median=3.5; freehand: median=3.5; media keys: median=3)

## 6.5 Focus

Looking at the logged data depicted in Figure 5, we can observe that participants were able to carry out the interaction without iTunes in focus with all modalities. Using the graspable interface or media keys, participants even carried out more commands without focus than with focus. Participants stated that they felt that they were able to carry out the commands without looking at iTunes when using the graspable (median=5) and touch (median=4). In contrast, they stated to prefer looking at iTunes when using the media keys (median=2.5, which contradicts the logged data) and freehand gestures (median=2.5). Concerning the latter, they stated that they miss the haptic

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Percentage of all commands carried out without iTunes in focus (not possible with the mouse)</figcaption></figure>
feedback, but some also said that it also felt “magical” and that they enjoyed observ-
ing the change in iTunes (e.g., the movement of the volume slider). This was not in-
tended by us, especially since we wanted to shift the device to the periphery of the
attention, but the novelty of the interaction seemed to have pushed the participants to
look at the visual feedback.

6.6 Feedback.

Due to the nature of our use case, functional feedback [31] was given. Users could hear whether a song stopped, started to play, was skipped or how the volume changed. Thus, participants stated that they did not miss further feedback for most devices (graspable: median=5; media keys: median=5; touch: median=4.5; freehand: median=3.5). Most participants (88%) named freehand gestures as the modality that offered the least feedback because of a lack of physical contact. Asked about the device with the most feedback, six participants (75%) named the graspable interface, because they could feel its physicality.

6.7 Location of Devices

Participants (all right-handed) were free to position the peripheral device on their desk wherever they felt comfortable. Only the media keys had their fixed locations on the keyboard. We asked them where they put each device while they had it at home.

*Graspable:* Five participants located the graspable interface on the right side of the keyboard. Two participants placed it on the left side and one participant changed the location several times. All participants interacted with the corresponding hand.

*Touch:* Three participants positioned the touch-sensitive device on the right. Three positioned it on the left. The respective hand was used for interaction. Two participants never used it on the desk, but carried it around, because the cable was only necessary for power supply and not for transmission. They interacted with the right hand. Other two participants also stated that they enjoyed the ability to take the device with them, even though we did not intend this use case.

*Freehand:* Seven participants positioned the freehand gesture tracking on the right side. With the exception of one, who used the left hand, they also interacted with the corresponding right hand. One participant changed the location frequently and thereby also the interacting hand.

*Media Keys:* Media keys were located on the bottom of the keyboard. Participants mostly used their right hand for interaction, but occasionally also used the left hand.

6.8 General Remarks

Generally, our participants considered the extension to be positive (median=4), espe-
cially because they were able to control iTunes independently of the current system
status (median=5). They considered the peripheral input fast.

Participants pointed out at least some weakness for every modality. The preproduc-
tion prototype for the freehand gestures was considered to be too big. The touch de-
vice had to be correctly oriented. In future implementations, this could be overcome by including the finger orientation [30], or by tilting the device a bit, as is already common with separated track pads. Participants considered grasping the knob to be more difficult with the non-dominant hand than touch or freehand interaction. Media keys ask for a very precise interaction. On the other hand, two participants mentioned that they liked media keys because they were integrated and no extra device was necessary. Others enjoyed the possibility of using touch remotely. Furthermore, they liked freehand interaction because it was considered futuristic and playful. The graspable knob was described as ergonomic and similar to the known mouse. Overall, our participants could imagine using a peripheral device for further commands like opening the player, activating shuffle or even other applications.

# 7 Interpretation & Discussion

In the following we will present interpretations of the results and propose considerations for future projects incorporating peripheral interactions and their evaluation.

## 7.1 Alternative Peripheral Interaction

Table 2 summarizes the results for the three peripheral devices (one ‘plus’ being a medium score here). This table confirms previous research based on graspable interfaces, but also shows that touch and freehand gestures are an alternative. This is the case especially when a graspable device is unsuitable, for instance, because there is no space to place it. Some current findings, like the missing haptic feedback for freehand interaction, might be less problematic once users are more accustomed to freehand interfaces and trust them more.

<table><caption>**Table 2.** Characterization of all peripheral modalities. One plus here is a medium score.</caption><thead><tr><th></th><th></th><th>Graspable</th><th>Touch</th><th>Freehand</th></tr></thead><tbody><tr><td rowspan="2">Usage:</td><td>Preference over mouse & GUI</td><td>+++</td><td>+++</td><td>+++</td></tr><tr><td>Little mental load</td><td>+++</td><td>+++</td><td>+++</td></tr><tr><td rowspan="2">Periphery:</td><td>Interaction without focus on GUI</td><td>+++</td><td>++</td><td>+</td></tr><tr><td>Inherent haptic feedback</td><td>+++</td><td>++</td><td>+</td></tr><tr><td>Learnability:</td><td>Easy familiarization</td><td>+++</td><td>+++</td><td>++</td></tr><tr><td>Location:</td><td>Preference for dominant hand interaction</td><td>++</td><td>++</td><td>++</td></tr></tbody></table>

## 7.2 Usage of Peripheral Devices

Participants issued fewer commands (including commands carried out with the mouse) during the media keys condition. We see two possible explanations for this observation. First, the peripheral devices (graspable, touch, freehand) could have been
more innovative for our participants and therefore triggered their play instinct, second, subjective data shows that media keys imposed more mental load on the participants, which might have kept them from interacting with the player although they would have changed the song if interaction was more comfortable. This is also in line with subjective findings where participants told us that they would like to have a faster interaction for iTunes and consider the distraction imposed through window switches as high. Consequently, we attribute the increased interaction with the peripheral devices to an easier and smoother interaction and think that it indicates that peripheral interaction can lower the barrier to start interaction. While a low barrier for interaction might not be extremely relevant for audio player control it might help for tasks that participants should care about but do not particularly enjoy (e.g., keeping their availability status up to date for colleagues in an instant messaging client).

For all peripheral modalities, we can observe significantly more interaction with the peripheral device than with the mouse. While we acknowledge that this might be partly due to the novelty effect and the fact that participants knew their interaction was logged and tried to (maybe unconsciously) please us, we still consider this a tendency towards a preference for peripheral interaction in contrast to traditional mouse interaction in this use case. This is supported by general research in peripheral interaction [17] and the fact that this effect arose in all modalities (graspable, touch, free-hand) also indicates that the design space for peripheral interaction can be extended beyond graspable interfaces.

## 7.3 Shift to the Periphery

Measuring how much a device or interaction style moves into the periphery is difficult and peripheral interaction researchers still struggle with it. Although in-situ deployments are feasible because they offer the necessary time for the users to get accustomed to the device, they also limit researchers in their observations. For example, it would not be viable to equip participants with an eye tracker in their home for several weeks to evaluate the visual periphery. As a weaker indicator of visual attention, we therefore used the input focus of the GUI.

We observed that participants carried out commands without the focus on iTunes with every modality, which strengthens our belief that the design space for peripheral interaction can be extended and indeed interaction in parallel to other tasks is possible. Furthermore, subjective results show that the graspable and touch modality were especially easy to be carried out in parallel. Physical feedback therefore seems to be helpful even (or particularly) when interacting with minimal attention in the periphery. In our use case functional feedback provided by the task itself (e.g., music started or stopped to play) also supported peripheral interaction.

## 7.4 Learnability

Usually when designing systems for peripheral interaction the interaction itself is very easy and straightforward (e.g. swiping up and down the touch surface or carrying out a flick gesture) and does not really have to be learned. Our participants also familiar-
ized themselves with each modality very quickly. On the other hand, when evaluating peripheral interaction, all prototypes always require time to get used to and to shift to the periphery [16]. Interestingly, our study suggests that it is not necessarily only the device itself that the user needs to get accustomed to (especially when they are generally used to an interaction style such as touch input), but also the general usage of the periphery alongside traditional input. One participant mentioned that he had more trust in peripheral interaction without looking at the GUI after he had been testing several devices. Another participant told us that he first had to break loose of habits like using alt+tab to switch to iTunes.

## 7.5 Location of Devices

Up to now, previous work on peripheral interaction did not consider the question which hand was used to carry out the peripheral interaction. Generally, we observed a tendency towards positioning and interacting on the right side, which was the dominant hand for all of our participants. This is an interesting observation because apart from a possible preference for the dominant hand, one can argue for either direction when considering a traditional desk setting. On the one hand, most users are used to moving the right hand away from the keyboard to interact with the mouse, and therefore might also feel more comfortable interacting with the peripheral device with their right hand. On the other hand, the mouse often occupies the right hand while the left hand would be free for other tasks. However, the latter did affect the decision where to put the device less than the preference for the dominant hand. For future systems it is feasible to offer peripheral interaction for the dominant hand if the primary tasks allows.

## 7.6 Paper Prototyping vs. Field Study

The paper prototyping study was very successful in terms of finding suitable interaction sets for all devices. Participants considered all gestures to be very intuitive and natural. At the end of the paper prototyping study, we asked the participants which modality they preferred. The most mentioned modality were freehand gestures, followed by touch gestures and graspable interaction. The participants of the field evaluation answered the same question after the eight weeks and the most mentioned device was the graspable interface followed by touch and freehand gestures. This clearly hints that while paper was great to test the general gestures, the preferences differed very much in the end. Possible reasons are that (1) technical issues (e.g. freehand gestures being the least reliable input modality) were not present with the paper prototypes, (2) haptic issues, which were mentioned several times, did not play such a big role when touching just paper and (3) the missing interactivity when using paper. Especially since most devices incorporating peripheral interaction are tested in the field, paper prototypes here have proven to be a good tool to test interaction mappings intended for the periphery even though during a paper study the periphery cannot be mimicked properly.
## 8 Limitations

There are some limitations inherent in the nature of our study. The biggest problem turned out to be the fact that the three peripheral devices used hardware in different development stages. The graspable knob is a commercial product with good haptic and ergonomic characteristics. The touch device was based on a mobile phone, which evokes different ideas than a track pad. The freehand tracking was based on a preproduction prototype, which had worse recognition rates than the other two devices for some commands (e.g., flicks (next/previous) were already very well recognized). Consequently, freehand tracking also resulted in more incorrectly interpreted gestures. This might also explain the higher frequency of freehand gestures compared to graspable interaction. We assume that this affected subjective ratings and favored the graspable interface. In other words, although touch and freehand gestures already seem suitable for peripheral interaction, in this exploration they might even be more suitable if implementation quality increases.

Further we only tested this in one use case (audio player control) with very distinct features (e.g. only input was evaluated, output and feedback was inherent to the task, i.e., music started to play). Furthermore we used one example implementation for each modality – thoroughly chosen based on a paper prototyping study – but of course these example do not completely cover the capabilities of each modality.

Additionally, unexpected events like holidays and job changes affected the individual times spent with each modality. Two participants also mentioned that they would have liked to have more time with each device to fully embrace it.

## 9 Conclusion and Future Work

We presented an eight-week in-situ deployment evaluating different modalities (graspable, touch, freehand) for peripheral interaction. We contribute a first exploration in the usage of touch and freehand gestures as a means for peripheral interaction. Our research confirms previous work relying on graspable devices but shows that similar results – smooth interaction intensively used without focus on the audio player itself – could be achieved with touch-based interaction as well as freehand gestures. This broadens the design space for peripheral interaction making room for other use cases, where graspable interaction might not be fit (e.g. in settings with high hygienic demands).

We further found that participants had a preference for the dominant hand, although the interaction itself (e.g. a flick gestures) could easily be performed with the non-dominant hand. This has not been addressed yet in peripheral interaction research but can act as guidance for future manual peripheral interaction.

Last but not least, for evaluation of peripheral interaction one can observe a tendency towards field evaluation. We managed to find suitable interactions by means of a paper prototyping study even though we were not able to mimic interaction in the periphery properly. By utilizing paper prototypes or similar techniques in the early
design process, only actual promising ideas need to undergo a field trial reducing the
resources needed overall.

Related work already shows diverse applications areas for peripheral interaction.
Further research in other, especially non-desk-based, scenarios is necessary. We
therefore aim at further case studies to substantiate the promising results of this study.
Additionally, as participants stated that they do not want an extra device for each
peripheral task, we imagine adapting peripheral touch input for touch pads, as well as
freehand interaction based on camera tracking with the help of built-in webcams.
Thus, the reduction of additional hardware and the trend towards miniaturization of
tracking devices (e.g. Leap Motion) will help to further spread the use of beneficial
peripheral interaction.

## Acknowledgments

This work is funded by the German state of Bavaria. We thank Sebastian Boring,
Sebastian Löhmann, Sarah Tausch, and Michael Winking for their valuable input
during this project. Further we thank everyone participating in our paper prototyping
study as well as the field deployment.

## References

1. Alonso, M.B., and Keyson, D.V. MusicCube: making digital music tangible. CHI EA, (2005), 1176-1179.

2. Altmann, E.M., and Trafton, G.J. Task Interruption: Resumption Lag and the Role of Cues. CogSci, (2004).

3. Andersen, T. Searching for music: How feedback and input-control change the way we search. Interact, (2005), 144-157.

4. Bailey, B.P. and Konstan, J.A. On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state. Computers in Human Behavior 22, 4 (2006), 685-708.

5. Bakker, S., Hoven, E. van den, and Eggen, B. Design for the Periphery. Eurohaptics, (2010), 71-80.

6. Bakker, S., Hoven, E. van den, and Eggen, B. Acting by hand: Informing interaction design for the periphery of people's attention. Interacting with Computers, (2012), 119-130.

7. Bakker, S., Hoven, E. van den, Eggen, B., and Overbeeke, K. Exploring peripheral interaction design for primary school teachers. TEI, (2012), 245-252.

8. Bakker, S., Hoven, E. van den, and Eggen, B. FireFlies: Supporting Primary School Teachers through Open-Ended Interaction Design. OzCHI, (2012), 26-29.

9. Baudel, T., and Beaudouin-Lafon. M. Charade: remote control of objects using free-hand gestures. Communications of the ACM 36, 7, (1993), 28-35.

10. Bi, X., Grossman, T., Matejka, J., and Fitzmaurice, G. Magic Desk: bringing multi-touch surfaces into desktop work. CHI, (2011), 2511-2520.

11. Butz, A., Schmitz, M., Krüger, A., Hullmann, H.: Tangible UIs for Media Control - Probes Into the Design Space, CHI Extended Abstracts (Design Expo), (2005), 957-971.
12. Döring, T., Kern, D., Marshall, P., Pfeiffer, M., Schöning, J., Gruhn, V., and Schmidt, A. *Gestural Interaction on the Steering Wheel – Reducing the Visual Demand*. CHI, (2011), 483–492.

13. Edge, D., and Blackwell, A.F. Peripheral tangible interaction by analytic design. TEI, (2009), 69-76.

14. Ferscha, A., Holzmann, C., and Resmerita, S. The Key Knob. ICDCS Workshops, (2006).

15. Hausen, D., Boring, S., Polleti, J., and Butz, A. Exploring Design and Combination of Ambient Information and Peripheral Interaction. DIS Work in Progress, (2012).

16. Hausen, D., Reducing Cognitive Load by Using the Priphery of our Attention. Workshop Designing for Cognitive Limitations. In conjunction with DIS, (2012).

17. Hausen, D., Boring, S., Lueling, C., Rodestock, S., and Butz, A. StaTube: Facilitating State Management in Instant Messaging Systems. TEI, (2012), 283-290.

18. Hazlewood, W.R., Stolterman, E., and Connelly, K. Issues in Evaluating Ambient Displays in the Wild: Two Case Studies. CHI, (2011), 877-886.

19. Jordà, S., Kaltenbrunner, M., Geiger, G., and Bencina, R. The reactable*. ICMC, (2005).

20. Kajastila, R. A. and Lokki, T. A Gesture-based and eyes-free control method for mobile devices. CHI EA, (2009), 3559-3564.

21. Kane, S.K., Avrahami, D., Wobbrock, J.O., Harrison, B., Rea, A.D., Philipose, M., La-Marca, A. Bonfire: a nomadic system for hybrid laptop-tabletop interaction. UIST, (2009), 129–138.

22. Kranz, M., Freund, S., Holleis, P., Schmidt, A., and Arndt. H. Developing Gestural Input. ICDCSW, (2006).

23. Mäntyjärvi, J., Kela, J., Korpipää, P., and Kallio, S. Enabling fast and effortless customisation in accelerometer based gesture interaction. MUM, (2004), 25-31.

24. Mark, G., Gudith, D., and Klocke, U. The Cost of Interrupted Work: More Speed and Stress. CHI, (2008), 107-110.

25. McFarlane, D. Comparison of four primary methods for coordinating the interruption of people in human-computer interaction. Human-Computer Interaction 17, (2002), 63-139.

26. Olivera, F., García-Herranz, M., Haya, P.A., and Llinás, P. Do Not Disturb: Physical Interfaces for Parallel Peripheral Interactions. INTERACT, (2011), 479-486.

27. Pirhonen, A., Brewster, S. and Holguin, C. Gestural and audio metaphors as a means of control for mobile devices. CHI, (2002), 291-298.

28. Pousman, Z., and Stasko, J. A Taxonomy of Ambient Information Systems: Four Patterns of Design. AVI, (2006), 67-74.

29. Strachan, S., Murray-Smith, R. and O'Modhrain, S. BodySpace: inferring body pose for natural control of a music player. CHI EA, (2007), 2001-2006.

30. Wang, f., Cao, X., Ren, X., and Irani, P. Detecting and leveraging finger orientation for interaction with direct-touch surfaces. UIST, (2009), 23-32.

31. Wensveen, S. and Djajadiningrat, J. Interaction frogger: a design framework to couple action and function through feedback and feedforward. DIS, (2004), 177-184.

32. Wiethoff A., Schneider H., Rohs, M., Butz, A., and Greenberg S. Sketch-a-TUI: Low Cost Prototyping of Tangible Interactions Using Cardboard and Conductive Ink. TEI, (2012), 309-312.

33. Wobbrock, J. O., Morris, M. R., and Wilson A. D. User-defined gestures for surface computing. CHI, (2009), 1083-1092.

34. Zheleva, E., Guiver, J., Mendes Rodrigues, E., and Milić-Frayling, N. Statistical models of music-listening sessions in social media. WWW, (2010), 1019-1028.