An Interactive Multimedia Environment through Artificial Emotion
Kenji Suzuki (*)(**), Antonio Camurri (*), and Shuji Hashimoto (**)(*) Laboratorio di Informatica Musicale,
DIST - University of Genova
Viale Causa 13, 16145 Genova, Italy
E-mail: {kenji, music}@musart.dist.unige.it(**) Dept. of Applied Physics, Waseda University
School of Science and Engineering
3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555
Tel:03-5286-3233, Fax:03-3202-7523
E-mail: {kenji, shuji}@shalab.phys.waseda.ac.jp

SummaryThis paper presents a system based on a generalarchitecture [1] to build emotional agents for real-time,interactive multimodal environments. The architecture
has been successfully employed in several scenarios and,in particular, in an art installation at “Arti Visive 2”, amuseal exhibition held in Palazzo Ducale, Genova. We
describe the emotional agent that we developed for themobile robot in the art installation, which interacts withhumans by different channels: visual, music, and “style”
of movement.

2. The case study: Art installation
at the "Arti Visive 2" museal exhibition
We successfully employed the agent architecture inseveral applications. In this paper we report ourexperience from the interactive art installation at "ArtiVisive 2", a museal exhibition held in Palazzo Ducale,
Genova in October 1998.

Keywords: Multimodal human-robot interaction, Artand multimedia, Emotional agent, artificial emotion
1. Introduction
The main goal of our work is to explore paradigms ofinteraction between humans and a robot in theframework of museal exhibitions, theatre, music and art
installations. In our case study presented in this paper,
we have a small mobile robot on wheels as an agent
capable of communicating by means of several channels,including sound and music, visual languages and its kindof movement (smooth/nervous, tailwagging, fast/slow,etc.). The agent-robot embeds a computational model ofartificial emotions, which is constructed by taking
advantages of the self-organization of a model of
improved Kohonen's Self-Organizing Map. The robot
freely tours in a museal exhibition area, as one of thevisitors, a sort of medium between humans and machines
living together in the exhibition area. Sensors allow him
not only to avoid collisions with people surrounding him,
but also to observe the artworks, the visitors, to interact
with them (see figure 1). This is an example ofinteractive
multimodalenvironment.
Interactive
multimodal environments are active spaces capable toobserve
users
and
toestablish
high-levelcommunications with them by means of human gestures,movement, speech and singing. At the same time, such
spaces allow users to get feedback in terms of visualmedia, sound and music. The software component of amultimodal environment can be partially realized as apopulation of communicating agents.

2.1 Description of art installation
The small robot on wheels is a Pioneer 1 robotic
platform by ActivMedia Inc, that we further equippedwith a camera, infrared localization sensors, local audio
system, and two wireless communication channels forboth audio and video signals. The robot freely tours in
the exhibition as one of the many people who frequent it,a sort of medium between humans and machines living
together in the exhibition area. Sensors allow him not
only to avoid collisions with people surrounding him, butalso to observe the artworks and the visitors to interact
with them. Figure 1 (right) shows the robot at work. Therobot has been "dressed" with a scenography for the art
installation. On the top of the dressed robot, a small
camera has been installed.
Visitors can communicate with the robot in several ways.
For example, they can approach it, act in front of its"eyes", follow it, ignore it, obstacle its path. The robot
interprets some stimuli as positive, other as negative
causing the evolution of its emotional state. As we saybelow, the emotional state is exhibited by means ofrobot's movement, music, sound, and visual media.
The robot integrates inputs from different channels. The
Saphira [2] low-level software of the robot gathers
information from robot's sensors every 100ms. From
Saphira data, the agent computes absolute and relative
positions of a human with respect to the robot, thedistance and the area occupied by the human near therobot. In this art installation we also use a camera toacquire a video image stream in front of the robot, towarp it on the basis of the current emotional state. Thedynamic video images warping is then projected on avideo screen visible from the public.
The system can control the robot by means of both highlevel behaviors (e.g., avoid obstacles, follow or escape ahuman), and simple movements. Movements andEncode Stimuli
High-level behavior

RATIONALINPUT

OUTPUT

Stimuli

Robot's Sensors
Camera System
Image Data
Gesture Recognition

EMOTIONAL
Emotional State

Robot Control
Music Process"MAX"

Competitive Model(Neural Network)

MIDI and Audio

Visual MediaREACTIVE

RobotTelevision"Emotional Mirror""Saphira"
Reactive robot control

Figure 1. Overall of one-way data flow at the developed system "emotional agent" (left)
and Interaction with visitors at museal exhibition "Arti Visive 2" (right)
behaviors can be modulated by parameters: in this way
the robot can use also movement to exhibit its emotional
state.
Music is also one of the most important ways ofcommunication. In this system, we adopted emotionalparameters to "modulate" in real-time a music score
skeleton, implemented as an Opcode Max patch. The
patch receives an array of emotional values, which
influence in real time the music generation algorithm.
In addition, output components include the visualcomponent based on the idea of "emotional mirror". The
robot sees what it is in front of it (people's faces,artworks) warped according to its emotional state. For
example, a face could appear "mirrored", distorted in avortex or re-processed with bright colors, respectively
corresponding to positive and negative emotional states.
During the performance, the system shows such images
on a TV screen in real-time.
2.2 The Model of Artificial Emotions
In this case study, the model of artificial emotion
embedded with robotic agent consists of four basic
emotions: the personality of the agent. These four states
are represented by a circle divided into four areas
differently colored. For simplicity, we named each statewith a typical human's basic emotional condition: Happy,Angry, Melancholy, and Tranquil. The details aredescribed in [3]2.3 Implementation
The system for the robotic agent works under Win32s
operating systems. We also adopted the Saphira software[2] developed at Stanford Research Institute, which
provides a behavior-level interface to control Pioneer 1
robot movements. The robot communicates by threedifferent radio links (digital I/O control data, video andaudio signals) with the supervisor computer on which themodel of artificial emotions is realized. The robot also
possesses an on board audio diffusion system, connectedby radio, which integrates the audio diffusion system

placed in the environment. Three computers connectedby an Ethernet network control various different aspects:
the first contains the emotional model and control
movement, the second deals with "emotional mirrors",
the third generates sounds and music. The robot agent
and the other applications are written in C++ (MS VisualC++).
3. Conclusion and future work
An important aspect in the developed emotional agent
concerns the effectiveness of multimodal communication
between the robot and humans: all the improvements
mentioned above should be carefully considered fromthis viewpoint of integration of different modalities [4].
In particular, the role of sound and music is a strong
concern in our work.
In conclusion, the emotional agent we have presented inthis paper allowed us to achieve some important results.
We hope that the ideas we sketched above could lead us
to other important results toward a natural and effective
human-machine interaction. The further consideration isto apply the emotional agent we developed for psychomotoric rehabilitation.
References[1] A. Camurri, A. Coglio "An architecture for emotional
agents", IEEE Multimedia, October - December 1998,
pp. 24-33 (1998)[2] K. Konolige, K. Myers, "The Saphira Architecture
for Autonomous Mobile Robots", Artificial
Intelligence and Mobile Robots, MIT/AAAI Press(1997)[3] K. Suzuki, A. Camurri, P. Ferrentino, S. Hashimoto"Intelligent agent system for human-robot interaction
through artificial emotion", in Proc. of IEEE Intl
Conference on System, Man, and Cybernetics, San
Diego, CA (1998)[4] J. Bates "The role of emotions in believable agents",
in Communications of the ACM, 37(3), pp. 122-125,(1994)