# Interactive Light Feedback: Illuminating Above-Device Gesture Interfaces

Euan Freeman, Stephen Brewster, Vuokko Lantz

► To cite this version:

Euan Freeman, Stephen Brewster, Vuokko Lantz. Interactive Light Feedback: Illuminating Above-Device Gesture Interfaces. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.478-481, 10.1007/978-3-319-22723-8_42. hal-01610841

HAL Id: hal-01610841

https://inria.hal.science/hal-01610841

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Interactive Light Feedback: Illuminating Above-Device Gesture Interfaces

Euan Freeman<fnref n="1" />, Stephen Brewster<fnref n="1" />, and Vuokko Lantz<fnref n="2" />

**Abstract.** In-air hand gestures allow users to interact with mobile phones without reaching out and touching them. Users need helpful and meaningful feedback while they gesture, although mobile phones have limited feedback capabilities because of their small screen sizes. Interactive light feedback illuminates the surface surrounding a mobile phone, giving users visual feedback over a larger area and without affecting on-screen content. We explore the design space for interactive light and our demonstration shows how we can use this output modality for gesture feedback.

**Keywords:** Above-Device Interaction · Gesture Feedback · Gesture Interaction · Interactive Light Feedback · Mobile Devices

## 1 Introduction

Above-device interfaces allow users to interact with their mobile phones using in-air hand gestures over the device. Gestures can be used when touchscreen input is unavailable or inconvenient; for example, users could gesture to follow a recipe on their phone without getting the touchscreen messy or having to wash their hands between using the phone and handling food. Gestures also allow more expressive forms of input than touch, allowing users to do more than simply touch the screen.

Users need helpful feedback when gesturing to be able to interact confidently and effectively. However, mobile phones have limited display capabilities because of their small size. Giving visual feedback about gesture interaction would affect on-screen content and may require redesigning application interfaces for gesture and non-gesture use. Audio and tactile feedback have been considered instead, although lacks the expressive, spatial characteristics of visual feedback. We propose giving visual feedback in the space *surrounding* the device, using light to illuminate surrounding table surfaces while users gesture (as in Figure 1). We call this *interactive light feedback*.

Mobile phones have already used light to illuminate table surfaces for other purposes. For example, Samsung's *Galaxy S6 Edge* [1] has a curved screen which illuminates the table surface when placed face-down, glowing different colours to show who is calling. Some of Sony's *Xperia* phones feature an *Illumination*

<footnote n="1">¹Glasgow Interactive Systems Group, University of Glasgow, Scotland
{euan.freeman,stephen.brewster}@glasgow.ac.uk</footnote>

<footnote n="2">²Nokia Technologies, Espoo, Finland
vuokko.lantz@nokia.com</footnote>
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Interactive light feedback around a mobile phone. Here, brightness changes dynamically to show how users how well their hands can be sensed.</figcaption></figure>

*Bar* [2] along one edge, which is used for visualisations during media playback. Research prototypes have also used lights embedded within the device bezel [8] and case [9] to visualise off-screen content. We build on these examples by looking at how interactive light can be used for dynamic gesture feedback instead.

In this paper we look at the design space for interactive light feedback. We share ideas about how light can be used to create gesture feedback and our demonstration allows others to experience these designs.

## 2 Related Work

Users require plentiful feedback when gesturing. However, mobile phones have small displays which limits the amount of feedback they can give. Others have suggested using other modalities instead for gesture feedback. Freeman *et al.* [4] looked at tactile feedback for in-air gestures. They evaluated a variety of methods of giving tactile feedback, including ultrasound haptics and vibration from smartwatches and other wearables. They found tactile feedback effective at enhancing visual feedback, although it lacked necessary bandwidth to replace it.

Others have used audio feedback for in-air interactions while phones are in a pocket, where visual feedback would go unnoticed. *Imaginary Phone* [6] and *Nenya* [3] read selected menu items aloud as users gestured, either in front of their body (for *Imaginary Phone*) or by moving a ring on their finger (for *Nenya*). In these cases, audio feedback came *after* gestures; users received no cues during interaction to help them. Our research looks at continuous interactive light feedback during gesture interaction, giving users cues as they interact.

In similar work to ours, Qin *et al.* [9] presented a prototype mobile phone which had LEDs embedded in its case. They presented an example application where users could respond to phone calls by touching the table on the left (illuminated red) or right (illuminated green) side of the phone. We build on this idea by using interactive light for dynamic feedback during gestures, rather than static feedforward before interaction. *Rainbowfish* [5] used a grid of LEDs to give feedback about gestures over a proximity-sensing surface, acting as a display. In our work, light is used to illuminate the area *around* a display, instead.
## 3 Interactive Light Feedback

Earlier we defined interactive light feedback as visual feedback given by illuminating the space surrounding a device. In our prototype (Figure 1), we use LEDs around the device edge to present interactive light feedback, although light sources could be enclosed within the bezel itself, as in *Sparkle* [8]. We use LEDs as they are small, cheap and have low power requirements. Our prototype consists of a flexible strip of 60 LEDs<fnref n="1" /> (approximately 7mm apart) affixed to a mobile phone and controlled by an Arduino microcontroller. Each LED is independently controllable and can vary its own hue and brightness.

Each LED has two design properties: **brightness**, which is always variable, and **hue**, which is fixed in some LEDs. These basic properties of light become more expressive when varied over **time** and **location**. Harrison *et al.* [7] showed how expressive and informative a single LED can be when change in brightness is animated over time. They presented a set of “light behaviours” which could be used to communicate information from a single light.

We have found dynamic change in brightness to be especially effective for gesture feedback, as users immediately see a response to their hand movements. One of our feedback designs uses brightness to show how well users’ hands and gestures can be sensed, using a visibility metaphor: when light is easier to see (that is, brightness is greater), it is because hands can be more easily seen (Figure 1). If light is less visible, it is because users are not as easily sensed; for example, their hands may be too far from the sensor.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Varying brightness at different locations to show hand position.</figcaption></figure>

Brightness can also be varied at different locations to create spatial patterns. For example, lights could be illuminated (that is, brightness greater than zero) to show hand position relative to the mobile phone (as in Figure 2). This is similar to the “shadow” feedback design in *Rainbowfish* [5], although feedback does not occlude on-screen content, since it is around the device instead. Spatial and temporal changes in brightness could also be combined to create rich feedback metaphors; for example, a progress bar could “fill up” over time as users dwell to make a selection (as with the gestures in [4]).

<footnote n="1">¹Adafruit NeoPixel LED Strip: http://www.adafruit.com/products/1506</footnote>
Hue can also be changed by some LEDs, allowing colourful feedback designs. Like brightness, hue can be varied over time and location to create expressive and informative feedback. Colours often have iconic meanings, making them useful for presenting state or static information. For example, green and red light could be used to show if a gesture was recognised or not. Hue could also be used for application-specific feedback metaphors; for example, using blue and red light to show temperature for thermostat control.

## 4 Summary

In this paper we described interactive light feedback, a technique which allows phones to give visual feedback about above-device gestures without affecting on-screen content. We discussed how brightness and hue allow rich information presentation and described how these could be used for gesture feedback. Our demonstration allows others to interact with a gesture interface and experience these designs themselves, to see how interactive light feedback can be used to informatively illuminate in-air interfaces.

## Acknowledgements

This research was part-funded by Nokia Technologies, Finland.

## References

1. Samsung Galaxy S6 Edge: http://www.samsung.com/global/galaxy/galaxys6/galaxy-s6-edge/ Accessed 10/3/15

2. Sony Illumination Bar: http://developer.sonymobile.com/knowledge-base/experimental-apis/illumination-bar-api/ Accessed 10/3/15

3. Ashbrook, D., Baudisch, P., White, S.: Nenya: Subtle and Eyes-Free Mobile Input with a Magnetically-Tracked Finger Ring. In: Proc. CHI '11. pp. 2043–2046. ACM Press (2011)

4. Freeman, E., Brewster, S., Lantz, V.: Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions. In: Proc. ICMI '14. pp. 419–426. ACM Press (2014)

5. Grosse-Puppendahl, T., Beck, S., Wilbers, D., Zeiss, S., von Wilmsdorff, J., Kuijper, A.: Ambient Gesture-Recognizing Surfaces with Visual Feedback. In: Proc. DAPI '14. pp 97–108. Springer (2014)

6. Gustafson, S., Holz, C., Baudisch, P.: Imaginary Phone: Learning Imaginary Interfaces by Transferring Spatial Memory from a Familiar Device. In: Proc. UIST '11. pp. 283–292. ACM Press (2011)

7. Harrison, C., Horstman, J., Hsieh, G., Hudson, S.: Unlocking the Expressivity of Point Lights. In: Proc. CHI '12. pp. 1683–1692. ACM Press (2012)

8. Müller, H., Löcken, A., Heuten, W., Boll, S.: Sparkle: An Ambient Light Display for Dynamic Off-Screen Points of Interest. In: Proc. NordiCHI '14. pp. 51–60. ACM Press (2014)

9. Qin, Q., Rohs, M., Kratz, S.: Dynamic Ambient Lighting for Mobile Devices. In: Proc. UIST '11 Posters. pp. 51–52. ACM Press (2011)