# Emotional Interaction with Surfaces - Works of Design and Computing

Larissa Müller, Svenja Keune, Arne Bernin, Florian Vogt

► To cite this version:

Larissa Müller, Svenja Keune, Arne Bernin, Florian Vogt. Emotional Interaction with Surfaces - Works of Design and Computing. 11th International Conference on Entertainment Computing (ICEC), Sep 2012, Bremen, Germany. pp.457-460, 10.1007/978-3-642-33542-6_52. hal-01556121

HAL Id: hal-01556121

https://inria.hal.science/hal-01556121

Submitted on 4 Jul 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Emotional Interaction with Surfaces - Works of Design and Computing

Larissa Müller<fnref n="1" />, Svenja Keune<fnref n="2" />, Arne Bernin<fnref n="1" />,<fnref n="4" />, and Florian Vogt<fnref n="1" />,<fnref n="3" />

**Abstract.** Here we present three interactive artworks that showcase the abilities of active textured surfaces to enable new qualities of surface interactions. Giving the surfaces different sensing abilities, ways to act and behaviors, then allows us to explore emotional attributes and character perceived by an observer. The three designs reflect artworks emotional perception of the observer and the resulting effects on behavior. The surfaces sense it's surrounding in terms of physical presence by proximity and emotional state based on facial expression. The surface can express its emotion with a change in movement and sound. The three surface designs explore further combinations of textile surfaces, sensors and reactive actuation to draw on the emotion of observers.

## 1 Introduction

Surface interfaces are predestined for manual and visual interaction, since they merge the modalities of physical input and visual output. Now, commonly used multi touch screens, particularly for mobile applications, enable physical interaction with planar rigid surfaces. Recent works have developed deformable interaction surface systems, which allow the incorporation of passive material attributes with surfaces in the physical interaction to enhance tactile feedback [1,2]. In our work we explore concepts of active surfaces, which are composed of interactive elements, inspired by repeating features in nature, in our case of animal scales, to find a trade-off between manufacturability and to form elaborate complex behavior of the overall surface structures. We see the potential to exploit these extended surface design spaces to form character, emotional attributes, and physical sensations to create compelling interaction.

We build on interactive works which combine emotional models and reactive objects based on facial [3], characteristic [4] and animalistic [5] physical artifacts. A number of emotional models have been developed to analyse human behavior and to create interactive systems. While in many existing works a strong association with prior experiences exists, we explore the mapping of behaviors. Less established associations are treated as degrees of freedom in a design space and let us guide by the affordances the designs provoke to find compelling mappings,

<footnote n="1">¹ Department Informatik, University of Applied Sciences (HAW) Hamburg, Germany</footnote>

<footnote n="2">² Department Design, University of Applied Sciences Hamburg, Germany</footnote>

<footnote n="3">³ Innovations Kontakt Stelle (IKS) Hamburg, TuTech Innovation GmbH, Germany</footnote>

<footnote n="4">⁴ School of Computing, University of the West of Scotland, UK</footnote>
with the goal to explore the ability of active surfaces to provoking and further respond to emotions of observers.

The works we developed are a part of an iterative process, based on the ideas from Mitch Resnick of a "'Lifelong Kindergarten'" [6]. A circle with the steps Imagine, Create, Play, Share and Reflect leads to fast prototypes. It also offers the possibility of creative working. The works we present consists of three different degrees of interaction. The first surfaces operates without any kind of sensor. The second one uses a distance sensor and the third is triggered by a visitors emotion, recognized from his face. We made tests with hundreds of visitors in which we were provoking some kind of interaction.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Emotional interaction triggered by: (a) random movements (degree one), (b) distance to the object (degree two), and (c) facial expression (degree three)</figcaption></figure>

## 2 Description of three Surface Designs

In the following we describe the three interactive artworks: Estoban, Mocoleme, and Lomelia that were created by the authors team with design and computer science backgrounds in terms of design concepts and interaction technology. Estoban and Mocoleme are focused on a relationship between sound atmosphere, movement and textile surface. Estoban affects the atmosphere in different urban spaces (private or public), and also has presence, the communication with Mocoleme is very simple. The intention is to have an affect on urban spaces and private rooms. Nature and urban sounds are calming, better than silence, which is often perceived as unpleasant and better for concentration or worklife than music. She reacts to distance with emotions, which are expressed by sounds and movement. Lomelia is based on a more complex kind of communication between an object and its observer. She includes the emotions of the observer to the interaction.
## 2.1 Estoban - Urban Stories

Estoban tells stories with sound. He creates a relaxed atmosphere and tells about his adventures from time to time. He shows his emotions in a subtle way. The border between reality and virtuality becomes fluently blurred. A direct contact to his surrounding is not what he looks for. He deals perfectly well with minimal attention. The attention received by listening to his stories is quite surprising.

We produced this first surface without any sensory input as an emotion backchannel. The displayed emotions are selected randomly. The output is using sound and movements to express and trigger emotions.

## 2.2 Mocoleme - Come Closer

Mocoleme likes to receive attention. If she feels neglected her mood changes quickly. She gives her opinions free running and is hardly affected. Once understood, you can play with Mocoleme in a wonderful way.

In this object we use distance as sensory input. The expression of emotions also appropriates sound and movements. The objective of this is to find out about preferences with respect to the input.

## 2.3 Lomelia - Emotional Dialogue

The emotions of an observer should not just be mirrored by the surface, but affect actions and emotions of visitors. It may get bored, for example, if a person is happy or be surprised following an anger outburst. She tries to frighten someone if the person is scared or gets annoyed by a lack of interest. The aim here is to express emotions by its movement.

The third surface, Lomelia, uses the SHORE library [7] from the Fraunhofer Institut of Integrated Circuits IIS as sensory input. This Software recognizes emotions from a visitors face. It gives a conclusion of the emotional states angry, happy, sad, or surprised. The camera placed on top of the surface is used to obtain an accurate picture of the current user. The servo motors are triggered by the assumed emotional states.

The overall system is separated into two modules with different functions: emotional display and emotional backchannel. The interfacing between these two modules is implemented via a central message broker that employs the publish-subscribe pattern. This enables the design to be flexible for further enhancements such as adding more modalities for input or output. The input module is implemented with infrared distance sensors and a camera that captures the faces of observers. The resulting images are emotion interpreted based on facial expressions implemented with the SHORE library [7]. The output module consists of a microcontroller to manage the servo motors of the interactive surfaces. All three active surfaces are constructed from fabric covered shell like base elements (Shown in Fig. 1c).
# 3 Summary

The three surface artworks demonstrates the concept of active surface to facilitate the possibilities of emotional interaction. The benefit of our interactive installations is that they are made of textiles. By an invisible technique the observer becomes focused on haptic allure, physical qualities and movements. The degree of emotional perception of the artworks exhibited here allow different forms to engage the observer. Informal exhibits of the artworks have provided much positive feedback, demonstrating that active surfaces can produce emotional experiences. The concepts presented here show that emotional awareness interfaces, inspired by human-human communication, with basic expressions can be highly engaging for participants. This technology could be transfered to entertainment computing through the exchange with researchers in the field.

## 3.1 Acknowledgement

We thank Kai von Luck, Renata Brink, Gunter Klemke, Birgit Wendholt and Franziska Hübler for their feedback and their support for this work. Further we express our gratitude to the members of our EmotionLab for supporting the construction of the artworks. In addition, we thank the Fraunhofer Institute of Integrated Circuits IIS for providing the SHORE library.

# References

1. Vogt, F., Chen, T., Hoskinson, R., Fels, S.S.: A malleable surface touch interface. In: Sketches and Applications at ACM SIGGRAPH. (2004)

2. Smith, J., Graham, T., Holman, D., Borchers, J.: Low-cost malleable surfaces with multi-touch pressure sensitivity. In: IEEE Workshop on Horizontal Interactive Human-Computer Systems, TABLETOP-07. (2007) 205–208

3. Becker, C., Prendinger, H., Ishizuka, M., Wachsmuth, I.: Evaluating affective feedback of the 3d agent max in a competitive cards game. In: International Conference of Affective Computing and Intelligent Interaction (ACII-05), Springer-Verlag (2005) 466–473

4. Breazeal, C.: Emotion and sociable humanoid robots. International Journal of Human-Computer Studies 59 (2002) 119–155

5. Wada, K., Shibata, T., Saito, T., Tanie, K.: Analysis of factors that bring mental effects to elderly people in robot assisted activity. In: IEEE International Conference on Intelligent Robots and Systems. (2002) 1152–1157

6. Resnick, M.: All i really need to know (about creative thinking) i learned (by studying how children learn) in kindergarten. In: Proceedings ACM SIGCHI Conference on Creativity & Cognition (C&C-07), ACM (2007) 1–6

7. Kueblbeck, C., Ernst, A.: Face detection and tracking in video sequences using the modified census transformation. Journal on Image and Vision Computing 24(6) (2006) 564–572