<figure><img src="image_1.png" /></figure>

HAL
open science

# Towards In-Air Gesture Control of Household Appliances with Limited Displays

Euan Freeman, Stephen Brewster, Vuokko Lantz

<figure><img src="image_2.png" /></figure>

## To cite this version:

Euan Freeman, Stephen Brewster, Vuokko Lantz. Towards In-Air Gesture Control of Household Appliances with Limited Displays. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.611-615, 10.1007/978-3-319-22723-8_73. hal-01610798

HAL Id: hal-01610798

https://inria.hal.science/hal-01610798

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Towards In-Air Gesture Control of Household
Appliances with Limited Displays

Euan Freeman¹, Stephen Brewster¹, and Vuokko Lantz²

¹Glasgow Interactive Systems Group, University of Glasgow, Scotland
{euan.freeman,stephen.brewster}@glasgow.ac.uk

²Nokia Technologies, Finland
vuokko.lantz@nokia.com

**Abstract.** Recent technologies allow us to interact with our homes in novel ways, such as using in-air gestures for control. However, gestures require good feedback and small appliances, like lighting controls and thermostats, have limited, or no, display capabilities. Our research explores how other output types can be used to give users feedback about their gestures, instead, allowing small devices to give useful feedback. We describe the *Gesture Thermostat*, a gesture-controlled thermostat dial which gives multimodal gesture feedback.

**Keywords:** In-Air Gestures · Household Devices · Multimodal Feedback

# 1 Introduction

Our homes are becoming increasingly interactive. Small household appliances, like lighting controls and thermostats, are being enriched with new interaction modalities. Users can now interact with their household environments using their smartphones, speech commands and even in-air gestures. Gesture interaction is especially compelling as users can interact from across the room and without having to first locate their smartphone or a remote control. Unlike speech, gestures do not disturb others and work in noisy household environments, where chatter, music and television may cause speech recognition difficulties.

In-air gesture interaction, like speech, can be uncertain and users require
lots of feedback during interaction, so that they can gesture confidently and
effectively. Many household appliances have limited ability to give feedback,
however. They typically have small displays, or no display at all, meaning visual
feedback would be difficult to see from a short distance away. Users would lose
the benefits of gesturing from across the room if they had to approach a device
first to see feedback about their gestures and the effects they are having.

Changing the form factor of such devices to accommodate a larger screen is undesirable, as doing so increases size, cost and power demands. In our research, we are exploring other ways for household controls to give users feedback during interaction. This paper describes the *Gesture Thermostat*, a gesture-controlled thermostat dial which gives users multimodal feedback about their gestures. Interactive lighting illuminates the surrounding wall to give visual feedback as
users gesture (see Figure 1). We also use sound, from within the device, and vibration, from a smart-watch, to enhance the feedback.

## 2 Related Work

Many have found that users are more willing to interact with small devices from a distance than they are to approach them for input. Valkkynen et al. [7] found that users preferred to interact with objects by pointing at them, rather than touching them, when more than a step away. Even when on their feet or walking past objects, users would rather gesture than go out of their way to touch them. Rukzio et al. [5] also found that distal interactions were more preferred than approaching objects to interact with them. These distal interactions allow what Koskela et al. [2] call *instant control* of household appliances: immediate, convenient access to basic functionality.

Offermans et al. [3] looked at interaction with household lighting. They found that access to controls had a strong influence on willingness to interact: users were more likely to interact when controls were easily accessible. They noted that mobile devices – remote controls and smartphones – often required too much effort and had inconsistent availability. We think in-air gestures could be a convenient alternative, allowing direct interaction when desired. Users could gesture from across the room for instant control of basic functionality, like switching lights off or setting the air conditioning to a cooler temperature.

In-air gestures are also available when other interaction modes are inconvenient or unavailable; for example, touch is less appropriate when cooking. *Kinect in the Kitchen* [4] explored this scenario further, allowing users to browse recipes and set cooking timers using in-air gestures. They found that gestures were effective but users required more feedback. Others have used in-air gestures for interacting with lighting controls, [1] and [6]. Both used the lighting itself as functional feedback: users saw immediate feedback from how the lights changed in response to gestures. Functional feedback is often unavailable and gives limited insight into interaction, however. Users receive little feedback during gestures and if movements are unrecognised, they receive none at all.

## 3 Gesture Thermostat

Gestures give users another way to interact with household devices, although they need sufficient feedback to help them interact. Many household appliances have limited display capabilities, which limits their ability to give feedback. Functional feedback is often unavailable and does not give users enough information to help them gesture. We are exploring other ways of giving users feedback during gesture interaction with household devices. We focus on interaction with thermostats as these are common devices with limited displays and no immediately noticeable effects; such devices benefit most from extra feedback.

Our *Gesture Thermostat* prototype (see Figure 1) has three types of output which can be used for gesture feedback: interactive light, sound and vibration.
Lights embedded in the thermostat dial (46 LEDs, approx 7 mm apart) illuminate the surrounding wall, giving visual feedback over a large area which users can easily see from across the room. An internal speaker provides audio feedback. Wearable devices, like smart-watches, could also be used as tactile displays for feedback. We use a simple actuator as a prototype smart-watch. In the rest of this paper and in our poster, we describe the *Gesture Thermostat*, its gestures and feedback designs, and initial findings from a user evaluation.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** The *Gesture Thermostat* illuminates surrounding areas for visual feedback.</figcaption></figure>

## 3.1 Gesture and Feedback Design

We use gestures for basic functionality – adjusting temperature – as this allows *instant control* [2]. More complex interactions, like programming the thermostat schedule, are less suited for in-air gestures. We give users two ways of adjusting temperature: (1) imprecise “punch” gestures, which increase or decrease temperature by a few degrees; and (2) precise “dial” gestures, which allow finer-grained temperature control.

Users initiate interaction by raising their hand with a closed fist. Upon detecting this gesture, the thermostat plays a short tone, emits a short vibration if a smart-watch is available, and reads the current temperature setting aloud; for example “twenty-two degrees”. All lights turn on with a low brightness and pulse slowly to show continued responsiveness. A single area of coloured light shows the temperature setting; blue light for cooler temperatures on the left (Figure 1, left), red light for warmer temperatures on the right (Figure 1, right).

From this position, users can quickly move their hand up-then-down or down-then-up to raise or lower the temperature, respectively. Audio and tactile feedback are given to confirm a recognised gesture and lights update to show the new thermostat setting. Alternatively, users can open their hand, as though about to grasp the dial, and turn it left or right to lower or raise the temperature. We use rate-based rather than position-based control due to hand instability and sensing inaccuracy. As users perform the dial gestures, white lights stop pulsing and the coloured area of light pulses instead; this helps users identify between interaction modes. Users are given feedback about their gestures, as before.
## 3.2 Initial Evaluation Results

Our initial evaluation asked eight users to complete tasks using the thermostat. Our findings suggest that our feedback designs support users effectively. Evaluation participants generally understood how feedback related to their hand movements and felt it made the device seem responsive; however, some wanted more information, especially about gesture sensing. When users encountered tracking issues, they were aware that something was wrong, but did not know why. More feedback about sensing could help users overcome these issues.

Although light feedback was effective on its own, users found it most salient and useful when paired with audio and tactile feedback. Light feedback was continuous which helped show responsiveness, while discrete audio and tactile feedback were easily noticed. Participants suggested that non-visual cues made visual changes more noticeable and made the thermostat seem more responsive.

# 4 Summary

In-air gestures allow users to interact with small household devices quickly and from a distance. Users need feedback to gesture effectively, although many household devices have limited feedback capabilities. Our *Gesture Thermostat* exemplifies how embedded lights, sound and vibration from wearables can be used to give gesture feedback. Our initial findings are positive: these outputs help users gesture effectively and make devices fun and responsive.

# Acknowledgements

This research was part-funded by Nokia Technologies, Finland.

# References

1. Djajadiningrat, T., Geurts, L., Bont, J.D., Chao, P.y.: Grace: A gesture-controlled wake-up light. In: Proc. DeSForM '12. pp. 130–136 (2012)

2. Koskela, T., Väänänen-Vainio-Mattila, K.: Evolution towards smart home environments: Empirical evaluation of three user interfaces. Personal and Ubiquitous Computing 8, 234–240 (2004)

3. Offermans, S.A.M., van Essen, H.A., Eggen, J.H.: User interaction with everyday lighting systems. Personal and Ubiquitous Computing 18(8), 2035–2055 (2014)

4. Panger, G.: Kinect in the Kitchen: Testing Depth Camera Interactions in Practical Home Environments. In: CHI '12 Ext. Abs. pp. 1985–1990. ACM Press (2012)

5. Rukzio, E., Leichtenstern, K., Schmidt, A.: Mobile Interaction with the Real World: An Evaluation and Comparison of Physical Mobile Interaction Techniques. In: Proc. AmI '07. pp. 1–18. Springer (2007)

6. Sørensen, T., Andersen, O.D., Merritt, T.: “Tangible Lights”: In-Air Gestural Control of Home Lighting. In: Proc. TEI '15. pp. 727–732. ACM Press (2015)

7. Välkkynen, P., Niemelä, M., Tuomisto, T.: Evaluating Touching and Pointing with a Mobile Terminal for Physical Browsing. In: Proc. NordiCHI '06. pp. 28–37. ACM Press (2006)