Proceedings of the 4th International Summer Workshop
on Multi-Modal Interfaces (eNTERFACE'08)

Christophe d'Alessandro

<figure><img src="image_1.png" /></figure>

To cite this version:

Christophe d'Alessandro. Proceedings of the 4th International Summer Workshop on Multi-Modal Interfaces (eNTERFACE'08). 98p, 2009. hal-01712604

HAL Id: hal-01712604

https://hal.science/hal-01712604

Submitted on 30 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
INTERFACE'08

PROCEEDINGS OF THE 4TH INTERNATIONAL SUMMER
WORKSHOP ON MULTI-MODAL INTERFACES

<figure><img src="image_3.png" /></figure>

LIMSI-CNRS
ORSAY, 4-29 AUGUST 2008

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

dépasser les frontières

<figure><img src="image_8.png" /></figure>
© CNRS-LIMSI, 2009

Printed in France

Distribution :

Available on order at

LIMSI-CNRS

BP 133

F91403 Orsay, France

Tel. 33 1 69 85 8X XX

Fax 33 1 69 85 80 88

[bibliotheque@limsi.fr](mailto:bibliotheque@limsi.fr)

Cover illustration: © Nathalie d'Alessandro

Edited by Christophe d'Alessandro

Audio & Acoustics Group, LIMSI-CNRS, BP 133, F91403 Orsay Cédex, France

[cda@limsi.fr](mailto:cda@limsi.fr)

[www.limsi.fr](http://www.limsi.fr)
EDITORIAL

In this digital age, Multimodal Interfaces are extending and replacing mechanical and electrical instruments. Impressive results have been achieved and new challenging applications are foreseen in fields like medicine, assistance to disabled persons, communication, design, performing arts, games, and many other types of human activity involving computer interaction.

Research on Multimodal Interfaces is a challenging interdisciplinary field, blending perception and action, sensory modalities and cognition, physics and the humanities, computer science and signal processing, theory and experiments, and last but not least science and art.

The summer 2008 has been a bit rainy in Paris, like it was in Mons in 2005, the ideal weather to welcome at LIMSI researchers in Multimodal Interfaces worldwide for the eNTERFACE'08 workshop, the 4th of a series of successful workshops initiated by SIMILAR, the European Network of Excellence (NoE) on Multimodal interfaces.

eNTERFACE'08 followed the fruitful path opened by eNTERFACE'05 in Mons, Belgium, and continued by eNTERFACE'06 in Dubrovnik, Croatia and eNTERFACE'07 Istanbul, Turkey.

The result was very successful, with 78 registered participants, working in 10 projects. It was also very international, as participants were coming from 20 different countries, and working in laboratories in 15 different countries.

The ultimate goal of this workshop was to make this four weeks event a unique opportunity for students and experts to meet and work together, and to foster the development of tomorrow's multimodal research community.

During the workshop we were delighted to welcome very distinguished speakers. We had the unique opportunity to listen to invited conferences and to discuss with the speakers in a friendly and studious atmosphere:

The 10 workshop projects explored many aspects of multimodal interfaces, including speech recognition and speech synthesis, embodied conversational agents, robots, haptic interface, various physiological signals, video cameras, body suit for motion capture. Smell and taste were the only missing sensory channels in these research projects. This was one of the main LIMSI staff's contribution, as we did our best to compensate this sensory deficiency by organizing barbecue parties

It is my pleasant duty to acknowledge the help and support of those people and institution that render this event possible.

The European Network of Excellence SIMILAR came to an end in 2007; this meant that we had no more financial support from SIMILAR for eNTERFACE'8. This year was the first post-SIMILAR workshop, with a different funding model. Launching out in this
project was challenging, because without the SIMILAR support it was difficult to guess
what would be the response of the multimodal research community.

In this new situation, we received the assistance and financial support of

* LIMSI-CNRS (for organization, rooms and facilities, equipment and support, network, and all on-site details)

* The OpenInterface Foundation for invited speakers, grants and room reservation (thanks to the UCL people under the supervision of Pr. Benoît Macq: Olga Vybornova, Jean Deschuyter, and Colin Michel).

* the COST2101 and COST2102 actions (European Cooperation in the field of Scientific and Technical Research) for student grants,

* the eNTERFACE'08 steering committee and particularly its President, Thierry Dutoit.

* and last but not least, most of the support came from the participants' organizations for travel and leaving expenses.

We gratefully thank all our speakers for sharing their time and ideas with eNTERFACE'08 people. All their contribution is available on the eNTERFACE'08 web site: http://enterprise08.limsi.fr/

Special thanks to those colleagues at LIMSI who committed themselves in eNTERFACE organization: Rami Ajaj, Albert Rilliard, Claudine Delcarte-Dang-Vu, Christian Jacquemin, Sylvain Le Beux, Nicolas Sturmel, Martine Charrue, Nadine Pain, Sophie Pageau-Maurice, Jean-Claude Martin, Nicolas Rajaratnam, Elisabeth Piotelat, Pierre Durand.

The invaluable and manifold help of Laurent Pointal is gratefully acknowledged.

And finally many thanks to all the participants, I hope you all had as much pleasure at working this summer at LIMSI as we had at organizing the event.

Chistophe d'Alessandro
Chair, eNTERFACE'08
# Implementing a Multiparty Support in a Tour Guide system with an Embodied Conversational Agent

Aleksandra Čereković (1), Hsuan-Hung Huang (2), Takuya Furukawa (2), Yuji Yamaoka (3), Igor S. Pandžić (1), Toyoaki Nishida (2), Yukiko Nakano (4)

(1) Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia, (2) Graduate School of Informatics, Kyoto University, Japan, (3) Department of Computer, Information and Communication Sciences, Tokyo University of Agriculture & Technology, Japan (4) Dept. of Computer and Information Science, Faculty of Science and Technology - Seikei University, Tokyo, Japan

## Abstract

In this project we aim to implement multiparty support in a Tour guide system with an Embodied Conversational Agent (ECA). We studied multiparty dialogue issues and adapted some of them to our system. As the outcome of the project, the first prototype of a tour guide system has been developed. The system can automatically detect presence of humans and invite them to a tour session. During the session, according to the number of users and their positions, ECA uses a gaze model implemented from conversational theory to address a specific user. ECA also reacts to decreased level of attention and tries to keep the interest in the session. Although conversational model in our system is rather simple, we find this work a first step for further system development in the ECA-human multiparty domain. Experiments with such systems can provide useful research studies in the field of interaction between ECA and humans.

**Index Terms**— multiparty communication, multiparty issues, ECA gaze model

## I. PRESENTATION OF THE PROJECT

### A. Introduction

Embodied Conversation Agents (ECAs) offer great promise to achieve natural interaction: they are anthropomorphic and have the potential to act like humans. Effects of ECAs are being studied through different roles they have in educational purposes, information delivery services, health care, sales and marketing. These studies indicate that ECAs can improve user engagement and significantly increase users' positive perceptions of learning experiences [10]. Despite all the positive effects, ECAs still mostly exist in research laboratories. As a recent state-of-the-art paper [14] claims, there is substantial research to be done on the way towards believable human-like ECAs.

So far, research community has mostly been oriented towards making ECAs capable of handling dyadic communication with a human or with another ECA. Proposed dialogue models have focused on monitoring just one conversational partner, which enables the ECA to participate in a “real” conversation. In multiparty ECA systems, most

issues belong to the dialogue management domain. In [9] Traum identifies those issues and groups them into three parts:

* **Participants' roles:** identification of local roles of participants which shift during interaction (addressee, listener, speaker), responsibilities in the dialogue, social roles

* **Interaction management.** Managing a communication flow in a multiparty system is far more difficult than in a dyadic system. Some of the difficult issues are how to give and recognize a turn and how to handle different channels (and backchannels). Besides, conversations can be easily split and merged together and attention can be paid to several persons.

* **Grounding and obligations** are notions commonly used to model local state of dialogue. In multiparty communication usage of these models can become very complex; e.g. if there are more addresses, it can become unclear what a proper grounding should be.

The issues identified by Traum have been used in the MRE project [12] for creating a negotiation model for virtual agents. Although this model is being used for several agents and one human, the ideas and factors presented can also be used as general guidelines for implementing multiparty support in ECA systems. Since this research was published after the workshop, it was outside the scope when developing the eNTERFACE '08 project.

Apart from Traum [1, 9, 12] there are relatively few researchers who studied conversation of multiple users and ECA. In [11] Rehm and Wisnner point out that “the literature on the behavior of multiple users is sparse”. Exception is the work from Vertegaal et al. [7] who studied gaze behavior shifts depending on conversational flow. They perform a real study on humans gazing in three-party conversation and propose a computational model of gazing which follows the findings from their study. They note that on average, subjects looked about 7 times more at the individual they listened to than at others and about 3 times more at an individual they spoke to than at others. They conclude that gaze, or looking at faces, is an excellent predictor of conversational attention in multiparty conversations. Rehm and Wissner [8] developed a
gambling system in which ECA plays a dice game with two humans. Game rules define the system scenario and make a simple dialogue model in which turn-taking mechanism and participants' roles are round-based. Their system lacks the active gaze model which follows human users. By using this system Rehm and Andre [11] investigated human gazing patterns in interaction with a real human and agent. Conclusions from their study are the same as Vertegaal's, however they also find that "People spent more time looking at an agent that is addressing them than at a human speaker." This phenomenon can be explained by the fact that prolonged eye contact in a social interaction can be found as impolite and rude; hence, the agent in this game might be considered an artifact and not a human.

Summarizing the state of the art, we conclude that the area of multiparty ECA systems is so far very little studied. A basis in communication theory has been set up by Traum, gaze behaviors have been studied in terms of conversational attention, but what lacks are systems which support it and which can serve as a test bed for identifying other issues or drawing interesting conclusions. In this project we aim to create a multiparty ECA system which will interact with two human users. Since we had a tour guide system at our disposal [6], we decided to use some of existing components (character animation player, platform) and investigate how the multiparty aspect can be incorporated into this system. Although it looked simple at first sight, multiparty communication support forced us to develop completely new components.

## B. Domains and challenges

In the system design phase, we planned it in such a way that it can recognize and respond to typical situations in a multiparty conversation. We added certain additional features we found mandatory or interesting also for dyadic communication.

### 1) *The multiparty aspect*

Since our research group lacks knowledge in natural language processing and understanding we decided to predefine a system scenario and predict users' behaviours. We chose a dialogue model in which the tour guide ECA has a role of narrator and keeps the initiative during the tour session. Narration is related to Dubrovnik's cultural heritage, history, and monuments in each of the five specific scenes we use in the system. During interaction users can interrupt the ECA and ask questions or just make a comment about the current topic. We predict topic-related questions and define it by using specific keywords in the speech recognition engine (such as "where, "when", "how"). To keep attention during the session ECA can also ask users simple "yes/no" answer-based questions.

Regarding the multiparty/multimodal aspect, we reference Traum's paper [9], focus on a limited number of issues and propose their solution in our system. Since our system doesn't use advanced dialogue models, there is no grounding, goals, or natural turn-management:

* **Appearance of users.** The system can dynamically recognize how many users are present and where they are standing. Besides, it can invite users to join the session. Depending on the number and positions of users, we use the agent's gaze direction to maintain the conversational flow. If there is nobody present, the system will be restarted.

* **Channel management.** We combine users' verbal channels with nonverbal behaviours (face orientation, gazing) to resolve users' utterance. However, the system can understand only a limited number of users' behaviours: decreased level of attention, making a request and leaving the system.

* **Speech collision.** During interaction it might occur that users ask questions at the same time. We handle this issue by using the agent to address one of the speakers and give him a turn, e.g. tour guide agent Dubravka says "You can speak one by one, I'll respond to you both. You can be the first one (addressing one user by pointing)"

* **Identification of conversational roles.** The system identifies local roles of participants during the session. Communication workflow is defined in the scenario; the agent keeps initiative and during the session he is the speaker most of the time, while the users are addressees. In the case when a user speaks, we identify him by using a localized microphone. It is a situation when the agent becomes an addressee, unless this user turns to another user and they start to communicate with each other.

Additionally in the multiparty domain we add two system features:

* **Handling users' mutual conversation.** This is a situation which we handle in "unnatural way". When users start to communicate with each other, the agent will try to get attention and continue his narration. Example is behaviour: "(waving) OK, let's continue with the tour". Since ECA can not understand the topic of their conversation we were forced to choose "a step back from empathy", but still retain ECA's politeness.

* **Inviting users to join the session.** The system can recognize human observers which are standing in the background and watching the interaction. They are treated as a group and, if present, the system will address them only if there are free places for users available. Example is the situation when user leaves the system and there is one person standing in the background. Then, Dubravka will look at the back and say "We have one free place, please come closer and join the tour".

### 2) Additional Features

We are also concerned with system features which can make interaction with the system fluent: the system can recover from failures, such as failure of speech recognition. In
case the user's speech is not recognized, we propose two-stage recovery. First, agent Dubravka will ask the user to repeat their question, and in case second failure occurs she will respond: "I'm sorry. I don't know a response to your question".

We also use the feature of Okao's vision [17] to detect gaze direction and recognize a situation in which the user is paying less attention to the session. If this occurs, agent Dubravka will speak until the end of her utterance, turn to the user and say "Seems you are not interested in this topic. Would you like to hear something else?"

### 3) Discussion

Issues addressed above are certainly not the only ones which must be solved to establish and maintain normal three-party interaction with ECA in a narration-based session. For this project these issues were general guidelines to implement a test bed for further research. After being integrated, it was planned to evaluate the system with random users (not during the workshop), and use the findings in further implementation.

### C. Overview of the work done

The work has been organized along the system design: system input, central part and system output. For mediating and synchronizing messages between system components in real-time we use GECA framework [5]. System requirements are mapped to component functions and during the workshop they are developed distinctly. At the beginning we had to reject some prior "ambitious" ideas (e.g. usage of BML [13] realizer which was supposed to be completed before the workshop). By the end of the workshop we didn't complete all plans; but we managed to integrate the first prototype of the tour guide system. The system is capable of running the first scene in which we aimed to handle all planned issues.

Input part is bimodal: it captures users' verbal and nonverbal behaviours (face orientation and gazing direction). For verbal input we produced a component which detects specific keywords and defines the user's preferences. As for nonverbal input we produced two components which are combined to resolve the following situations: users' arrival and departure, numbers of users present, situations in which users are paying less attention. Unfortunately, one month of the workshop wasn't sufficient to complete a tracking algorithm which was mandatory to detect a situation when users are starting their own conversation (they rotate their faces to each other and activate verbal channels). Also, we didn't have time to deal with image processing and divide the space to distinguish between the users and audience. This issue turned out to be rather complex because the area where the system was installed wasn't large enough (we couldn't distinguish between users and audience members by using only size of the face since there was no space behind the users). That's the reason why we allow a maximum of two users in the system's scope.

To manage input multimodalities and choose an appropriate agent's response we initially intended to use MIDIKI's toolkit [4]. Idea was to integrate multiparty support

in MIDIKI's state update mechanisms: e.g. nonverbal events and specification of gazing. During the workshop we realized that toolkit's documentation was inadequate for fast implementation and in the end we adapted an existing scenario component [6]. However, not all issues in the central part are implemented: it lacks speech collision handling and there is no smart scheduling and behaviour control (e.g. continuous control of gazing).

In the animation player we implemented patterned gaze behaviour towards users. Patterns follow findings from computational theory [7, 11]. We also modelled behaviours (gestures, facial expression) to increase ECA's believability. Animation player also lacks some features. We are not handling users' comments at the moment because there is no smart control for this level. We handle interruptions by deleting all queued behaviours and waiting for new behaviours from the central part of the system.

In the remainder of this paper, we describe the overall system. First, we depict and explain hardware configuration, while in the third chapter we explain system design and components' functionalities in detail. For the end, we discuss the system, evaluation and conclude the work completed on eNTERFACE '08 workshop.

## II. HARDWARE CONFIGURATION

System was installed at LIMSI, which was venue of eNTERFACE '08 workshop. It is depicted on Fig. 1.

<figure><img src="image_9.png" /><figcaption>Figure 1. System installed at LIMSI</figcaption></figure>

Two equal microphones are installed on two machines under the same settings. During installation we had to make sure that distance between microphones was great enough to avoid speech overlapping between microphones. Results of several experiments showed that Loquendo ASR [21] was stable enough not to recognize speech from approx. 0,5 m between users. We positioned a table between users to make sure they are not standing next to each other (Fig. 1).

System captures users' nonverbal behaviours by using two cameras; one sends data to Motion Detector which detects motions in the system's scope, while the other is connected to
<figure><img src="image_10.png" /><figcaption>Figure 2. System design</figcaption></figure>

Okao's vision component which can recognize face orientation and gaze direction if face doesn't exceed 60 degrees relative to the position of the camera (more details about how these components work will be presented in the third section.) To handle data from cameras we use another computer.

Scenario component runs on the third machine. It does not inquire any additional hardware, and neither does GECA server.

For system output we use another machine, projector and a large panel to display the agent and its behaviours. We calibrate the window position to make the agent large enough to make the conversation circumstances more natural (Fig.1)

### III. SYSTEM DESIGN AND COMPONENTS

System design is depicted in Fig. 2. System features explained in the first section are granulated and mapped into components which are divided into three categories: input, central part, and system output. All designed components are connected to GECA platform which can transport and mediate messages in real time.

#### A. System Input

Input data processed within the system are human speech, appearance, faces and direction of gazing. Combination of this data can resolve one the following situations:

* User arrival

* Number of users

* Interruptions

* Unrecognized speech

* Decreased level of attention

* Leaving the system

We also planned to detect and handle speech collision, conversation between users and presence of the audience (to invite them if there is a free place in the system). However, there was no time to complete the tracking algorithm and divide the space. Speech collision detection issue is simple

from the system input point of view, but takes more effort on the central part.

#### 1) Speech Recognition

Speech recognition component is developed with Loquendo ASR [21]. Experiments performed at the very beginning of the workshop showed that Loquendo is very satisfying for our needs; it is stable in noisy environments, speaker-independent, there is no significant speech overlapping at reasonable distance between users (0,5 meters approx.), and no keyword spotting issue, which was a problem with Microsoft SAPI. Besides, it has garbage-rules definition to reject words that are not defined in vocabulary, confidence score recognition and timing specification of recognized words, which are additional features used in our project.

Once the speech recognition component is started, it waits for users' speech input. When it receives a speech start signal, it starts recognizing the speech. The system also recognizes who is speaking. The results of recognizing the speaker and speech content are sent to OpenAirServer. The process flow is as follows.

1. Once the Speech Recognition component is started, this module starts listening to the audio stream.

2. When there is an utterance input from the user, this module gets the start time of utterance from Loquendo, as well as the utterance duration.

3. Finally, it transmits the results obtained in flow 2 as character string. It repeats flow2 and flow3.

SR sometimes reacts to voices which are not from the user. We think such an error can be prevented by using a Speaker Verification function in Loquendo.

#### 2) Nonverbal Behavior Recognition

To detect nonverbal behaviours we developed two components: Okao Vision and Motion Detector. In cooperation these components have two functions: they can
detect the number of users and calculate the user's attention level towards the agent.

<figure><img src="image_11.png" /><figcaption>Figure 3. Detection of user's face and estimation of gaze directions with Omron's Okao Vision</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 4. Input image processing with Motion Detector</figcaption></figure>

Okao Vision is a commercial product made by Omron [17]. It is a library that provides accurate face detection and extra functions like face orientation, gaze direction, the positions and openness of eyes and mouth, gender detection, age identification and face identification from one single image. Fig.3 is an example output of a program written with Okao Vision. Requirements and limitations vary with functions. Face detection requires a 10x10 pixel square image of the face and the head rotation must be within 60 degrees in left-right direction (yaw) and 30 degrees in up-down direction (pitch). Face orientation and eyes-mouth openness detection require a 40x40 pixel image, and the rotation of the head must still be within 60°(yaw) and 30°(pitch). Gaze direction requires an 80x80 pixel image and is limited to 30°(yaw) and 15°(pitch). In our test program with a 960x720@15fps web cam, accuracy of face detection is sufficiently high and undoubtedly usable, but most of the other functions are not

reliable and can only be treated as an extra bonus. Since Okao Vision does not require stereo cameras, it is an acceptable result. We thus decided to use the face detection, face orientation and eye-mouth openness. Face orientation is also used to approximately determine the users' gaze direction. This is not very accurate but should be sufficient if we only need to distinguish rough directions like the screen or another user.

Motion Detector is a component that can detect a moving object in distinct areas, and is developed by using OpenCV library [20]. The component divides the viewed region into two distinct areas and detects motions inside each area. Fig. 4 shows video image data divided into two areas surrounded with a blue and a red square. Moving objects are recognized when the sum of differences in pixel values between the current and previous frame is above a certain threshold.

To determine the number of users we use both Okao Vision and Motion Detector. Okao Vision has a function to determine the number of user faces, but it cannot detect a user's departure because it fails in face detection when rotation of user head exceeds 60 degrees. Hence, we use Motion Detector to track movements in the area.

The number of users should be equal or less than two and they stand at positions decided beforehand. Therefore we can know the user existence in the area and the number of users by detecting the movement in region by using Motion Detector. The result of this function is the greater of the values for the number of people provided by Motion Detector and OkaoVision, respectively. This function has been implemented and tested.

Although the topics that the users are talking about serve as a very important cue for knowing whether the users are interested in the system, it is not practical to use speech recognition in our noisy real-world application. Therefore, we hypothesized that users' interest can be approximated as the users' attention toward the system during a relatively long interval of time, such as dozens of seconds. Attentions of the users can be approximated as the patterns at which users are looking at a high ratio of time. The meaningful patterns we consider are either the agent or other users. The attention of the users toward the system or the agent can also be discovered by tracking whether there is a change in the users' activities immediately after a remarkable action done by the agent.

## B. Central Part

### 1) Nonverbal Input Understanding

This component combines input data coming from Motion Detector and Okao's Vision and uses simple heuristic methods to resolve the number of users and level of interest in the system. As Okao's Vision fails in detection when users rotate their head beyond 60 degrees, it is important to save data in short time periods and combine it with information from Motion Detector. For example, two users talk/listen to the agent and the left user turns his head to see who entered the room behind him. In this situation Okao sends value of `UserNumber` variable set to one, as if there is just the right
user in the system. At the same time, Motion Detector detects motions in the left area, and Nonverbal Input Understanding component sets `UserNumber` value to two and sends it to the Decision Making Component.

## 2) Decision Making Component

At the beginning of the workshop we planned to implement a central part following the Information State Theory [2] [3], which is a general theory of human-agent dialogues. A dialogue based upon this theory is defined as a set of variables (or information) that describe the current state of the dialogue. According to the modeled dialogue, the information state is updated by dialogue moves that compose the agent's and the user's utterances. The agent's dialogue strategies which trigger particular dialogue moves in response to current conditions that are caused by both of the user's and the agent's dialogue moves are defined as plans. The agent then behaves as a regular rational autonomous agent that chooses and executes plans to achieve its goals. This is the general idea of the dialogue theory; dialogue moves, dialogue plans and lexicons are user-defined and require intensive work. The prototype was being implemented based on one of the implementations of information state dialogue move engine [4] to be capable to deal with multi-modal, multi-party conversations, dynamically changing behaviors accompanying the emotion dynamics simulating component, etc.

However, due to lack of time to complete the dialogue manager by the end of the workshop, instead of a fully functional dialogue manager, a power-up version of existing GSML (Generic Embodied Conversational Agent Scenario Language) [5] script interpreter was developed.

Compared to the previous system which merely matches recognized speech inputs and non-verbal inputs with predefined patterns, a variable system is introduced. Following information state theory, interaction between the agent and one of two users is described with a set of variables like a snapshot. For example, `SpeechInput` represents the most recent result from one of the speech recognition components, `Speaker` represents the id of the speech recognition component, `UserNumber` represents the number of users who are standing in the user area, `UserStatus` represents the arability of the users, `UserAttention` represents how much the users are paying attention to the system, `Addressee` specifies who should be the addressee of the agent's next utterance, etc.

The values of these variables are updated with the agent system's internal status and perception events sent from the speech recognition components and non-verbal input interpretation component. How the value of the variables should be updated can also be specified by the script designer in the script as the effects of particular input patterns. `Effect` element is introduced into `Template` element for this purpose. An input event can cause the values of particular variables to be bound to, added with, or subtracted from certain values.

The syntax of the patterns defined in GSML scripts is also extended. `Predicate` element is introduced to represent a

test on the values of a variable. The value of the variables can be tested to be equal to, lesser or greater than certain values.

The chatbot-like ECA system is then extended to a more powerful rule-based autonomous system. The agent or the script execution engine updates its internal status variables via the perceptions from the outside world or the users, and picks the first valid template which made all of the conditions (predicates) true to perform. Therefore, the rules like that the tour guide agent should walk to the front to greet when there are users present in the user area, say goodbye to the user and go back to the initial position when the user has left the user area and so on can be specified in the script.

States limit possible patterns that will be used in matching in the current conversation situation and thus isolates the interference from other states which may happen to have the same triggering patterns. Because of the absence of context management mechanism in the agent's behavior control, there is no way to determine whether a user answer is related to the last question asked by the agent. However, for example, when the agent is going to ask a yes/no question like "Do you need a tour guide? ", a transition to a specific state representing the question can isolate the question from other yes/no questions.

`GlobalState` is introduced for error and interruption handling. When a failed or unknown recognition occurs, appropriate response will be searched among the categories defined in the global state. When interruptions from the user like "excuse me" or "pardon" occur, they are also matched with the patterns defined in this state.

Unlike a full dialogue-managing central component, the disadvantage of this approach is: the agent does not conduct a plan that contains multiple steps to achieve a certain goal. The agent's behaviors are driven by the events that occurred in the outside world. The management mechanism for information like grounding or topics is not included in the script execution kernel. These features are still implementable but are left as script programmer's responsibility.

During the period of eNTERFACE'08 workshop, only a few issues specific to multi-party conversation could be addressed. Because it is possible that there can be two users standing in front of the tour guide agent, the gaze direction of the agent becomes essential in three-party dialogue. Because we do not do natural language processing and have no way to understand the meaning of the conversation, our simplified model is: the agent should distribute different ratio of its gaze depending on who the addressee is. It will look at both users at the same frequency or look at the user standing at the left hand side more frequently if he is the addressee. Detailed gaze control is not done by the central component but is by the player. An `Addressee` attribute is introduced in the `Utterance` element to specify the addressee of this utterance is to the left, right or both. Addressee specification is done by the rules in the scenario script by the script programmer; for example, this utterance of the agent should be directed towards the last speaker and so on.
### C. Animation Player

The component which displays the city of Dubrovnik and tour guide agent Dubravka was developed by using Visage SDK tool from Visage Technologies company [15]. As this product does not have support for virtual environments, we used a normal image as background. Position of the agent was calculated by using ARToolkit software from eNTERFACE '06 workshop [6]. Size of the agent was adjusted to large in order to efficiently address the user by gazing (Fig. 5).

<figure><img src="image_13.png" /><figcaption>Figure 5. Example of agent's behaviours in Animation player</figcaption></figure>

Agent's behaviors defined in system scenario are produced and tested by using GSML syntax [5] which synchronizes nonverbal behaviors (gestures, locomotion, facial expressions) with speech. Animations for the character were either modeled manually in 3ds Max or procedurally in Visage.

Gazing model runs on the utterance level and is controlled by the central part of the system according to the *Addressee* attribute. E.g. if the addressee is the left user, the agent will gaze first at the left side, then it will glance to the right for a while (if there are two users) and gaze back to the left user again. Since we cannot predict the duration of the utterance, we repeat patterns and stop when the utterance is finished.

Animation Player receives messages from Decision Making component and runs them in an appropriate way. We distinguish between four types of messages:

* **Utterances.** Contain GSML behavior description to be performed by an agent. After they have been received, they are put into queue.

* **Interruptions.** Messages which stop the agent from performing "planned" behaviors. When received, the behavior queue is cleaned.

* **Stop messages.** Force player to shut down and save current content.

* **Reset messages.** Reset the player to the initial state. This message is received when there are no users within the system scope.

It was also planned to add message priority within Utterance category to avoid unnecessary re-planning in case of interruptions like comments. Example is a situation when the agent is speaking and the user says "That's very nice". From the system input it is recognized as interruption, however in system output we treat it in this way: the agent stops with speech, responds to the user ("Yes, it is"), and then continues with the behavior which was running before the comment occurred. A level of "smart scheduling" according to the utterance priority should be achieved for this feature.

## IV. DISCUSSION AND CONCLUSION

### A. Summary of software produced.

A first prototype of ECA tour guide system which can communicate with two users has been developed. The system supports some of the multiparty aspects that were obtained from the literature and planned from the beginning of the workshop. The system can:

1. Recognize presence of people and invite them to the session

2. Resolve the number of users and their level of attention

3. Handle interruptions, such as the users' requests, and respond to these requests by using the agent's gaze direction

4. Recover from failure (e.g. if a user's speech isn't recognized)

5. Recognize users' departure and automatically finish the session.

It was also planned to have a system with five different scenes, but as writing a scenario and manual modeling of the agent's behavior was rather time-consuming, we focused only on first scene. Extension of the system on five scenes doesn't require modification of existing components- it is enough to write a new script and, if necessary, add new animations to the player.

### B. Towards system evaluation

Since some of the planned multiparty aspects were not yet implemented and tested, up to now no system evaluation has been performed.

After we have completed our work, the first thing to do is to see how our system acts in normal interaction with one and two humans (what lacks at the moment is that system doesn't handle users' mutual conversation). As it cannot "understand" natural language, we are in particular interested in learning what type (and level) of interaction can be achieved with this scenario. Of course, we want to improve it, if necessary and possible. Compared to the previous dyadic tour guide system which was only served for demo purposes, this system is several steps ahead.

On the system-level, evaluation involves three things: evaluation of user-ECA performance, e.g., how fluent and efficient it is, evaluation of the user experience from a
subjective standpoint and evaluation of the effectiveness of the ECA-enabled application in achieving its goals [19]. At this point we are interested in the first and second type of evaluation. Usually, results of experiments are collected by using questionnaires or monitoring users' bio-signals in interaction, such as gazing or speech. Overviews of relevant factors which influence human-character interaction are given in [18], and will serve us as general guidelines to set up an experimental system and choose an appropriate system study.

Compared to dyadic ECA systems our system has one strength from evaluation point of view: it provides a possibility to measure and compare human-human interaction and ECA-human interaction at the same time. Although the full system does not “allow” communication between humans, measurements can be simplified by observing attempts of humans to establish communication and neglect ECA. It would also be interesting to compare evaluation results to the recent work by Rehm [16], who performs evaluation study on an emotionally tense lying game. The agent in this system is even less “smart” than ours (it can understand some digits and “yes/no”-variants as well) and is treated in three different ways by human users: as a normal conversational partner, as an interlocutor and as an artifact. This is, however, expected, but the question is; what are actually variations among these categories and what are their values? What is the level of empathy and effectiveness which an ECA has to have to keep attention, and what is the duration of this attention? How to model an ECA which can be “interesting” for prolonged communication? Our system can serve as a useful test bed for further studies.

### C. Conclusion

Implementing multiparty support in ECA systems partly relies on communication theory which has set up several bases (e.g. [9]). However, systems which can establish and maintain multiparty communication with two humans are scarce. During this workshop we attempted to implement a first prototype of our system which handles this type of communication. Although some of the tasks were not completed, first experiments in interaction with the system are encouraging. For example, the system can detect the number of users, distinguish local roles of users in a conversation and use ECA's gazing direction to specify an addressee.

The weakest point in the system is the conversational model which is rather simple, but overall outcome of this workshop is just a step towards further experiments and studies in the field of multiparty communication in ECA systems.

### ACKNOWLEDGEMENTS

This research is partially supported by the Ministry of Education, Science, Sports and Culture of Japan, Grantin-Aid for Scientific Research (S), 19100001, 2007, "Studies on Construction and Utilization of a Common Platform for Embodied Conversational Agent Research" and the Ministry of Science Education and Sports of the Republic of Croatia, grant nr. 036-0362027-2028" Embodied Conversational Agents for Services in Networked and Mobile Environments." We would also like to thank Dr. Lao Shihong from

OMRON Corporation Research & Development Headquarters, regarding licensing of OKAO Vision.

### REFERENCES

[1] D. Traum and J. Rickel, "Embodied Agents for Multi-party Dialogue in Immersive Virtual Worlds," in AAMAS 2002, vol. 2.

[2] Traum, D., Bos, J., Cooper, R., Larsson, S., Lewin, I., Matheson, C., and Poesio, M. (1999). A model of dialogue moves and information state revision.

[3] Larsson, S., Berman, A., Gronqvist, L., and Kronlid, F. (2002). TRINDIKIT 3.0 Manual. Trindi Deliverable D6.4

[4] The MITRE Corporation (2005). Midiki User's Manual, version 0.1.3 beta edition.

[5] Huang, H., Cerekovic, A., Pandzic, I., Nakano, Y., and Nishida, T.: The Design of a Generic Framework for Integrating ECA Components, Proceedings of 7th International Conference of Autonomous Agents and Multiagent Systems (AAMAS08), Estoril, Portugal, pp128-135, May, 2008.

[6] Huang, H., Cerekovic, A., Tarasenko, K., Levacic, V., Zoric, G., Pandzic, I., Nakano, Y., and Nishida, T.: An Agent Based Multicultural Tour Guide System with Nonverbal User Interface, the International Journal on Multimodal Interfaces, Vol. 1, No. 1, pp. 41-48. 2007.

[7] Vertegaal, R., Slagter, R., van der Veer, G., and Nijholt, A. 2001. Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, United States). CHI '01*. ACM, New York, NY, 301-308

[8] Rehm, M., André, E., and Wissner, M. 2005. Gamble v2.0: social interactions with multiple users. In *Proceedings of the Fourth international Joint Conference on Autonomous Agents and Multiagent Systems (The Netherlands, July 25 - 29, 2005). AAMAS '05*.

[9] David Traum. 2003. Issues in multi-party dialogues. In *Advances in Agent Communication (F. Dignum, ed.). Springer-Verlag LNCS*.

[10] Lester, J.C. and Converse, S. A.: The Persona Effect: Affective Impact of Animated Peda-gogical Agents. In S. Pemberton, ed. Human Factors in Computing Systems: CHI'97 Con-ference Proceedings, pp. 359-366. New York: ACM Press (1997)

[11] Matthias Rehm and Elisabeth Andre. Where do they look? Gaze Behaviors of Multiple Users Interacting with an Embodied Conversational Agent. In Proceedings of Intelligent Virtual Agents (IVA), 2005.

[12] David Traum, Stacy Marsella, Jonathan Gratch, Jina Lee, and Arno HartHolt. Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-Modal Virtual Agents. In Proceedings of Intelligent Virtual Agents (IVA), 2008.

[13] Kopp, S., Krenn, B., Marsella, S., Marshall, A., Pelachaud, C., Pirker, H., Thórisson, K. and Vilhjálmsson, H.: "Towards a Common Framework for Multimodal Generation: The Behavior Markup Language", in Proceedings of the IVA '06 conference, Marina del Rey, USA, 2006

[14] Vinayagamoorthy, V. and Gillies, M. and Steed, A. and Tanguy, E. and Pan, X. and Loscos, C. and Slater, M. (2006) *Building Expression into Virtual Characters*. In: Eurographics 2006, 04-08 Sept 2006, Vienna, Austria.

[15] http://www.visagetechnologies.com

[16] Rehm, M. 2008. "She is just stupid"-Analyzing user-agent interactions in emotional game situations. Interact. Computing. 20, 3 (May. 2008), 311-325

[17] http://www.omron.com/r_d/coretech/vision/okao.html
[18] G. Ruttkay, Z. Pelachaud, C., 2004. From Brows till Trust: Evaluating Embodied Conversational Agents. Kluwer Academic Publishers, Dordrecht / Boston / London.

[19] Evaluating Embodied Conversational Agents" seminar, 15-19 March, 2004 Organized by Z. Ruttkay, E. André, K. Höök, W. Lewis Johnson, C. Pelachaud (http://wwwhome.cs.utwente.nl/~zsofi//eeca/“)

[20] http://sourceforge.net/projects/opencvlibrary/

[21] http://www.loquendo.com/en/

**Aleksandra Čereković** is a Ph.D. student at the Department of Telecommunications, Faculty of Electrical Engineering and Computing in Zagreb, Croatia. She obtained her diploma degree on audio-visual speech processing (lip-sync) in year 2006 from the same University. Currently she is working on topics of body animation and control for Embodied Conversational Agents (ECAs). Her research interests are: body communication, gesture theory, animation modeling, and computational learning.

**Hung-Hsuan Huang** graduated from the Computer Science Department of National Chen-Chi University, Taiwan in 1998 and obtained his master degree of computer science and information engineering from National Taiwan University, Taiwan in 2000. After a two-year military service where he was the political warfare director and the co-commander of an army company, he went abroad to Japan. After the learning in a Japanese language school for one year, he entered the Ph.D. course of the Graduate School of Informatics of Kyoto University, Japan in 2003. His research interests include intelligent software agent, information visualization, photo management, gesture interface.

**Takuya Furukawa** (Japan, October 1984) graduated from the Human and AI Systems Department of Fukui University, Japan in 2007 and entered the master course of the Graduate School of Informatics of Kyoto University, Japan in 2007. His research interests include multimodal conversational interfaces, and human behaviors in communication with embodied conversational agents. He has an experience of fellowship job in Works Applications is Japanese ERP software company. Mr. Furukawa is a member of JSAI (Japanese Society for Artificial Intelligence).

**Yuji Yamaoka** is a student of a Master course program of Ubiquitous and Universal Information Environment, at the Graduate School of Informatics, Tokyo University of Agriculture and Technology, Japan. His research interests include culture in nonverbal behaviors of ECAs.

**Toyoaki Nishida** is a professor of Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University. He received the Doctor of Engineering degree from Kyoto University in 1984. His research centers on artificial intelligence and human computer interaction. In 2001, he founded a series of international workshops on social intelligence design (see http://www.ii.ist.i.kyoto-u.ac.jp/sid for more details). Then, he broadened the scope of research to include understanding and augmenting conversational communication, and opened up a new field of research called Conversational Informatics. Currently, he leads several projects on social intelligence design and conversational informatics. He is a member of the board of directors of IPS (Information Processing Society) of Japan and JSAI (Japanese Society for Artificial Intelligence). He serves as an editorial board member of several academic journals, including Web Intelligence and Agent Systems, AI & Society, and Journal of JSAI (editor-in-chief).

Associate Prof. **Igor S. Pandžić** received his BSc degree in Electrical Engineering from the University of Zagreb in 1993, and MSc degrees from the Swiss Federal Institute of Technology (EPFL) and the University of Geneva in 1994 and 1995, respectively. He obtained his PhD from MIRALab, University of Geneva, Switzerland in 1998. In the same year he worked as a visiting scientist at AT&T Labs, USA. In 2001-2002 Igor was a visiting scientist in the Image Coding Group at the University of Linköping, Sweden, and in 2005 visiting researcher at Kyoto University, Graduate School of Informatics, Nishida & Sumi Lab.

He is now an Associate Professor at the Department of Telecommunications, Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia. His main research interests are in the field of computer graphics and

virtual environments, with particular focus on facial animation, embodied conversational agents, and their applications in networked and mobile environments. Igor also worked on networked collaborative virtual environments, computer generated film production and parallel computing. Igor was one of the key contributors to the Facial Animation specification in the MPEG-4 International Standard for which he received an ISO Certificate of Appreciation in 2000.

Asst. Prof. **Yukiko Nakano** received her bachelor degree in psychology from Tokyo Women's Christian University, Japan in 1988 and one master degree in educational psychology from the University of Tokyo and one in media arts and sciences from Massachusetts Institute of Technology, USA in 1990 and 2002, respectively. She obtained her Ph.D. in information science and technology from the University of Tokyo, Japan in 2005. Yukiko was a researcher of NTT Research Laboratories from 1990 to 2000 and was a sub-leader researcher of Research Institute of Science and Technology for Society from 2002 to 2005. She was working at the Department of Computer, Information and Communication Sciences of Tokyo University of Agriculture and Technology as an associate professor from 2005 to 2008. From summer 2008 she has been working at Dept. of Computer and Information Science Faculty of Science and Technology, Seikei University. With her special interest in Embodied Conversational Agents (ECA), she has been studying human face-to-face communication in psychology and communication science, and creating multimodal conversational interfaces based on a model of human communication behaviors.
# Multimodal High Level Data Integration

Olga Vybornova, Hildeberto Mendonça (1), Daniel Neiberg (2), David Antonio Gomez Jauregui (3), Ao Shen (4)

(1) UCl-TELE, Louvain-la-Neuve, Belgium, (2) TMH/CTT, KTH Royal Institute of Technology, Sweden, (3)

TELECOM and Management SudParis, France, (4) University of Birmingham, UK

## Abstract

A method of multimodal multi-level fusion integrating contextual information obtained from spoken input and visual scene analysis is defined and developed in the context of this research. The resulting experimental application poses a few major challenges: the system has to process unrestricted natural language, free human behaviour, and to manage data from two persons in their home/office environment. The application employs various components integrated into a single system: speech recognition, person identification, syntactic parsing, natural language semantic analysis, video analysis, human behavior analysis, a cognitive architecture that serves as the context-aware controller of all the processes, manages the domain ontology and provides the decision-making mechanism.

**Index Terms**—multimodal data integration, semantic representations, cognitive architecture, spoken input, video input, decision-making.

## I. PRESENTATION OF THE PROJECT

### A. Motivation

Since Bolt's groundbreaking work on speech- and gesture-based interaction [2], multimodal systems and fusion engines have progressively evolved towards more robust semantic interpretation and they all include some fusion mechanism to combine the separate data streams by reducing uncertainty. The interest to jointly process audio and visual information related to human activities, and to extend the technological developments in individual modalities for human-computer interaction has been increasing. To improve naturalness and robustness of user interfaces they should be able to automatically recognize human identity, intentions, speech, actions in computing environments, define the person localization, perform source separation, media synthesis and media content mining. A critical factor that allows to make multimodal processing in speech-based interactions effective is the robust integration or fusion of information from multiple modalities.

Multimodal fusion is a central question to solve and provide effective and advanced human-computer interaction. To understand and formalize the coordination between modalities involved in the same multimodal interface it is required to extract relevant features from signal representations and thereafter to proceed to high-level (semantic-level) fusion. Signal- and semantic-level processing is deemed tightly interrelated since signals are seen as bearers of the meaning.

To provide natural and robust interaction with the user(s) the system must be able to interpret as a whole the various modalities used by a human to interact, i.e., to extract information arriving simultaneously from different sources and to combine them into one or several unified and coherent representations of the user's behavior. A challenge is to fuse these different input modalities effectively to augment their natural strengths and use redundancy to increase robustness. Moreover, it is critical to decide what information should be fused and what should not.

We see multimodality as a set of variables describing states of the world (user's input, an object, an event, behavior, etc.) represented in different media and through different information channels. In this respect the goal of data fusion (merging semantic content from multiple streams) should give an efficient **joint interpretation** of the multimodal behavior of the user(s) – to provide effective and advanced interaction.

We have been working with multimodal fusion considering two main modalities, which are speech recognition and human behavior analysis through image processing tools. We have conducted experiments in a hypothetical context, with two or more people interacting with each other and with objects in the scene. We observed that when intending to perform an action, the user(s) might:

* speak about it describing their actions or intentions (the ideal case, then the linguistic and action recognition data are complementary);

* be doing things, but speaking about something absolutely irrelevant to those actions (in this case the data from the two modalities should be analysed separately, without merging);

* the users' words might contradict the actions performed (then the actions have priority, since "actions speak louder than words");

* be just silent when performing actions (in this case the unimodal operation mode is required and no data fusion is needed).

Everything what is said or done is meaningful only in the particular context. To accomplish the task of semantic fusion we should take into account the information obtained at least in the following three types of context [7]:

- domain context: meaning prior knowledge of the domain, semantic frames with predefined action patterns, user profiles, situation modelling, a priori developed and dynamically updated ontology defining subjects, objects, activities and relations between them for a particular person;
- conversation context: derived from natural language semantic analysis;

- visual context: capturing the user's gesture/action in the observation scene and allowing eye gaze tracking to enable salience models.

Basically, in our approach the high level fusion of the input streams can be performed in three stages: (i) early fusion that merges information already at the signal or recognition stage to provide reliable reference resolution, (ii) late fusion that integrates all the information at the final stage to give interpretation of the human behavior [26], and (iii) reinforcement fusion that execute one or more cycles of verification, reevaluating all identified meanings to improve the overall integration. We argue that in order to make the multimodal semantic integration more efficient and practically real, special attention should be paid to the stages preceding the final fusion stage.

During the early fusion, some data received from each modality, which meaning is redundant in two or more modalities, should be synchronized in order to strengthen the link between data and optimize the late fusion. An example of early fusion done in our experiment is the person identification from the speaker and from the image features. This singular meaning helps us to complement a recognized intention from the speech with the behaviour of the speaker, predicting what decision is more appropriated for the situation.

When the early fusion finish, the search for semantics continues with the application trying to identify plurality of meanings. This is the input of the late fusion, which deals with higher symbolic elements abstracted by the modality recognizers. These elements are first mapped between the different modalities and then integrated during the final decision stage to resolve uncertainty and produce the user's intention interpretation. One example of plurality of meanings on speech recognition is to detect the human intention linking the person (subject) with a concrete or abstract things (object) through a action or relation (predicate). In the sentence "I want to call Nick" there is a clear intention represented by links between "I", "want to call" and "Nick", which are subject, predicate and object respectively. These meanings will be combined with other meanings during the late fusion.

In addition to the early and late fusion, we are contributing with the *reinforcement learning*, which is a cyclical process that uses all found meanings to reinforce or redefine them selves, producing better inputs for the late fusion. The need for reinforcement came from a reflection about how a existent modality can help the analysis of other modalities preserving the modality cohesion, where a modality must not be aware of the presence of other modalities in order to avoid some level of coupling. We could observe this need in our experiment, when we implemented n-best reranking functions for the speech recognition analysis to find the best hypothesis about what was said. These functions need direct access to the list of concepts from the ontology and what was analysed from other modalities, in order to probabilistically rank the best hypothesis possible. The reinforcement fusion is more robust than early fusion, less conclusive than late fusion, and it is not

mandatory, but recommended when there are clear possibilities of improvement.

Developing optimal strategies for fusion is still a crucial issue as none of the known approaches has so far emerged as the outstanding solution. Various different formalisms have been proposed and mainly consist of deterministic fusion schemes:

* rule-based implementations: including variants such as unification grammars used in the multidimensional parsing of typed feature structures, finite state transducers supporting tighter coupling of parsing and input recognition, finite state machine for biologically inspired rhythmical synchronization;

* statistical methods: such as integration based on maximum entropy;

* mixed approaches: probabilistic agent-based resolution and integration performed with simple voting mechanism based on confidence values of resolved concepts, probabilistic salience weighting of action-based concepts that are bound to speech concepts according to a predefined set of rules.

## II. APPLICATION, SCENARIOS AND MODALITIES OF INTEREST

In order to develop and validate the whole experiment, we defined five different scenarios where two people interact mutually in a room, talking in a natural way and behaving without restrictions. Each scenario tries to explore specific combinations of speech and behavior to increase the robustness of the system. This system is able to track people, analyze their behavior, movements, speech, and takes decisions about how to prompt them necessary information when required or provide any other assistance.

The main challenges that we face in this application are unrestricted human language and free natural behavior within home/office environment. In order to analyze behavior efficiently, the system has to correctly process, interpret and create joint meaning of the data coming from speech analysis and video scene analysis. We consider that human behavior is goal-oriented, so our main aim is to recognize the users' plan, to see what they want to achieve.

The system manages the data streams arriving from two sources – video scene and speech. In particular, we show a technique distinguishing between the data from different modalities that should be fused and the data that should not be fused but analyzed separately.

## III. PROJECT ARCHITECTURE AND IMPLEMENTATION

The application employs various components integrated into a single system: speech recognition, speaker identification, syntactic parsing, natural language semantic analysis, video analysis, human behavior analysis component and a cognitive architecture that serves as the context-aware controller of all the processes, manages the domain ontology and provides the decision-making mechanism.
Figure 1 below describes how all components collaborating mutually when connected to compose the overall solution. The process starts when an audio or a video signal is detected by the first time. There is no restriction about what signal should start first because all modalities can be processed in parallel and independently. When an audio stream is received, the speech recognition component processes it, generating a string of what was said. The same signal is sent to the speaker identification component, which will associate what was said with who said that. The string is sent to the syntactic parsing component to identify the syntax of each word, which is important for the natural language semantic analysis component, responsible for the identification of the subject, the agent, the predicate, the object of interest and other elements. From the semantic analysis, it is possible to extract semantic structures very similar to the structure of the knowledge base, represented by ontology. If we find the identified semantic in the ontology then it means that sentence is valid inside of the context and can be useful to fuse with other meanings coming from other modalities.

<figure><img src="image_14.png" /><figcaption>Figure 1. Project architecture</figcaption></figure>

On the other side, when a video stream is detected, the image is processed by image processing component called Open CV. This component analyses some image features to calculate the position of each person on the horizontal plan of the scene, their movements direction and identifies who is each person according to a predefined profile. It is also important to identify people through this modality because we have to know the position of who is talking in order to associate the user intention with his actions. The synchronization of detected users in each modality is done during the early fusion.

The next step is to analyse the human behaviour, comparing the movements of the user with a set of rules. The behaviour is relative to fixed objects in the scene, which are defined in the context domain and are directly associated with the aid to be given by the system. The rules define the boundaries of what is near of far from a certain object. Then the result of the rule processing at this stage is: [person] is near to [telephone], [person] is far from [computer] or [person] is moving to

[library]. This result is produced for each person in each frame of the video. Individually, these results are not significant enough for fusion. We have to analysis the movements in many frames in order to have final conclusions. For instance: if in the last 80 frames the rule engine produced “[person] is moving to [library]” then we can conclude that there is a real intention to achieve the library, considering some variables of the environment, such as area of the room. The late fusion occurs when we identify a person moving to the library and we also detect the intention to find a book in a sentence like this: “I can find a book about it in the library.”. Therefore, we can conclude the person is moving to the library because he wants to find a certain book there. Previous sentences, already analysed, indicates that the article “it” in the current sentence actually means “French wines”, an object identified through the semantic analysis.

Many existent components were just reused, such as Sphinx, C & C Parser, C & C Boxer, Open CV, Protegé and Soar. Other components were developed from the beginning, which was the case of the Human Behavior Analyzer, the Fusion Mechanism and the interoperability between all components, which is implemented using sockets, a simple network communication strategy.

## Speech Recognition

### 1) Speech Data

The speech was recorded by two non-native subjects, one 23 year Chinese male and one 32 year Swedish male. The data was recorded using 16kHz, 16 bit audio. The 5 scenarios consisted of a total of 72 sentences and 148 seconds. For development of speaker identification, we used ten phonetically rich sentences for training, and for parameter tuning we used another ten phonetically rich sentences and ten words of different length.

### 2) Speech Recognition

For speech recognition we used Sphinx 4, which is an open source, Java-based speech recognizer [29]. For acoustic modeling, we used the 8 Gaussian triphone models, trained on the Wall Street Journal Corpus, which are supplied along with Sphinx. Since we wanted to allow the system to monitor a discussion between two or more people, we want to have a large vocabulary language model. For this purpose, 3-grams with a maximum of ~5000 words were trained using the orthographic transcriptions from the Wall Street Journal Corpus. The 5000 words were selected as the most common ones plus the ones that are present in the scenarios.

In order to enhance understanding of concepts, we wanted to evaluate different methodologies to pick hypotheses from the N-best list which Sphinx can produce. For this purpose, Sphinx was configured to produce as long N-best lists as possible given a memory constraint of 1 Gb. This resulted in list lengths of ~100 hypotheses for short utterances with good accuracy and ~10000 list lengths for long utterances with poor accuracy. Two different approaches may be used. The first is to estimate a re-scoring function on data, the second is to apply re-ranking heuristics. The first approach is likely to be most efficient, but it requires enough data to be done
optimally. The second approach is easier to apply and is likely be more robust to less knowledge about the users may say.

By making an assumption about the domain, a word list consisting of all nameable things in the ontology was queried. Three kinds of re-ranking heuristics were tried out. In each of these, a score were assigned to each hypothesis and then N-best list was re-sorted using this score in such way that the initial ranking was preserved if two or more hypotheses had equal score.

**Method A:** The score is set to one if one or more words from the ontology are found.

**Method B:** One scoring point is added for each matching word in the ontology word list.

**Method C:** A score is added, which is equal to the word length in characters, for each matching word in the ontology word list.

In Tables 1 and 2, the performance gain by this approach is compared to selecting the 1st top hypothesis and by letting an oracle select the hypothesis which maximizes Accuracy. These results where obtained using a preliminary version of the ontology. While Accuracy takes substitutions, insertions and deletions into account, Correctness does only take substitutions and deletions into account. In Table Y, Accuracy and Correctness is computed by only using the words from the ontology as a reference, forming Concept Accuracy and Concept Correctness. The idea is to examine to what extent information, which the cognitive modules can process, is present.

<table><caption>Table 1: Word accuracy and correctness for different hypothesis selection methods</caption><thead><tr><th>Hypothesis selection method</th><th>Accuracy</th><th>Correctness</th></tr></thead><tbody><tr><td>1<sup>st</sup></td><td>56,30</td><td>64,52</td></tr><tr><td>Method A</td><td>50,13</td><td>62,98</td></tr><tr><td>Method B</td><td>46,53</td><td>63,50</td></tr><tr><td>Method C</td><td>47,81</td><td>64,01</td></tr><tr><td>Oracle</td><td>74,81</td><td>78,15</td></tr></tbody></table>

<table><caption>Table 2. Concept accuracy and correctness for different hypothesis selection methods</caption><thead><tr><th>Hypothesis selection method</th><th>Accuracy</th><th>Correctness</th></tr></thead><tbody><tr><td>1<sup>st</sup></td><td>54,12</td><td>67,06</td></tr><tr><td>Method A</td><td>27,06</td><td>68,24</td></tr><tr><td>Method B</td><td>10,59</td><td>78,82</td></tr><tr><td>Method C</td><td>12,94</td><td>81,18</td></tr><tr><td>Oracle</td><td>78,82</td><td>74,12</td></tr></tbody></table>

From the results presented in Table X, it is clear that the methods don't improve over the baseline (1st hypothesis). It should also be noted, that these results are obtained using a preliminary version of the ontology. From the results in Table

Y, it should be noted that while there is a large drop in Accuracy, there is a clear increase in Correctness, which means that insertions are present. This means that the information content in terms of words known to the ontology is increased on the expense of word insertions.

### 3) Speaker Identification

Speaker Identification is the task to determine who is speaking. For the application is described in this report, a standard speaker identification system was considered. It is based on Mel-Frequency Cepstral Coefficients (MFCCs) and Gaussian Mixture Models (GMMs), as in Renynold et. al. We used 28 log-Mel filters between 300 and 8000 Hz, cosine projected to 24 dimensions, where the first 12 with their delta where used. A simple data driven procedure for speech detection was tried: The MFCCs where clustered using the k-means algorithm with euclidean distance for two clusters. Then the cluster which had the highest energy was marked as "speech" and the other one as "non-speech". Then these two clusters where used for frame based segmentation. Visual inspection showed that the approach seemed reasonable. No channel compensation was used. This system was implemented in Matlab.

The two speakers in the scenarios where enrolled, using ten phonetically rich sentences. For parameter tuning, another ten phonetically rich sentences and ten words were used. The performance measured in classification accuracy for various numbers of Gaussians are shown in Table 3. Further analysis showed that the errors occurred for test utterances containing very short words, such as "no" and "hi". Using speech detection didn't improve accuracy. Based on these results, 16 Gaussians where chosen for evaluation using the recorded data for the scenarios. The accuracy for the evaluation data was 94%.

<table><caption>Table 3. Speaker identification accuracy, with and without speech detection (SD)</caption><thead><tr><th>Gaussians</th><th>No SD</th><th>SD</th></tr></thead><tbody><tr><td>2</td><td>85,00%</td><td>92,50%</td></tr><tr><td>4</td><td>90,00%</td><td>92,50%</td></tr><tr><td>8</td><td>97,50%</td><td>92,50%</td></tr><tr><td>12</td><td>97,50%</td><td>92,50%</td></tr><tr><td>16</td><td>97,50%</td><td>90,00%</td></tr><tr><td>20</td><td>97,50%</td><td>95,00%</td></tr><tr><td>24</td><td>92,50%</td><td>95,00%</td></tr><tr><td>32</td><td>97,50%</td><td>92,50%</td></tr></tbody></table>

### 4) System integration of the speech components

A speech module, consisting of Sphinx 4 and the Speaker Identification system described above was fully implemented
in a TPC/IP client. Also on the server side, the re-ranking
method A was implemented, in way which makes it easy to
extend to Method B or C.

B. *Syntactic and Semantic analysis of spoken input*

The next stage after speech recognition is syntactic and
semantic analysis of the discourse. For our purposes we use
the CCG (Combinatory Categorial Grammar) parser, Release
0.96, developed by S. Clark and J. Curran [9]. The grammar
used by the parser is taken from CCGbank developed by J.
Hockenmaier and M. Steedman [3].

CCGbank is a treebank containing phrase-structure trees in the Penn Treebank (WSJ texts) converted into CCG derivations. It allows easy recovery of long-range dependencies, provides a transparent interface between surface syntax and underlying semantic representation, including predicate-argument structure. The grammar is based on 'real' texts, and that is why it has wide-coverage, thus making parsing efficient and robust.

The CCG parser has Boxer [8], as add-on to generate semantic
representations - Discourse Representation Structures (DRSs),
the box representations of Discourse Representation Theory
(DRT) [13]. DRSs consist of a set of discourse referents
(representatives of objects introduced in the discourse) and a
set of conditions for these referents (properties of the objects).
Our initial experiments with the CCG parser together with
Boxer showed that it suits well for our application in parsing
speech of elderly people which is on one hand somewhat
restricted to their environment, background and social
relationships (the domain model and user profile help us cope
with the challenge), but on the other hand is naturally broad
enough to need wide-coverage tools for processing. DRSs can
be generated in different output formats in Prolog or XML. To
link the DRSs output with the ontology we use the XML
format.

C. Visual scene analysis

1) Introduction

The video information provides a description of what happens
in the environment at a given time. In this project a video
sequence is made for each possible scenario. The video
sequences were recorded using a distributed 8-camera
voxelised visual hull [22]. The description of the environment
is obtained by processing each image of the videos using
computer vision algorithms.

2) Problem

In these video sequences, there are 3 types of fixed objects (a
telephone, some books and a computer) located in different
positions inside the scene, there are also two persons that are
moving and interacting with these fixed objects. In order to
have a good description of the environment for each one of
these scenarios is necessary to extract the information of the
position of each fixed object, the position of each person at
any moment and also the motion direction of each person.

Using this information it's possible to know if a person is near
or far from each object or if a person is moving to an object.
By this way, the system will have good information about the
human behavior in order to make better decisions.

3) Procedure

Extracting all this information of the video sequences are
common problems of computer vision field. These problems
are mainly:

* Object detection (to find the position of each fixed object)

* People detection and tracking (to find the position of each person)

* Motion analysis (to find the motion direction of each person)

The computer vision system to extract this information was
implemented in C++ using OpenCV library [12] developed by
Intel. In the next sub-sections, will be described the procedure
and algorithms to resolve the problems already mentioned.

a) Finding the position of each object

The objects (the telephone, the books and the computer) in
these video sequences are always fixed (in the same position),
also they don't change size or rotate because the camera's
viewpoint is always the same. Hence, the detection of these
objects can be easily done by using a template matching
algorithm [7]. These algorithms compare a template with a
region of an image in order to determinate a similarity
measure, wherein the similarity measure is determined using a
statistical measure.

To obtain these templates, a sample picture of each object is
captured from any image of the video sequence. The OpenCV
operator cvMatchTemplate was used, this operator returns the
probable positions in the image where the template can be
located. By this way, the most probable position corresponds
to the area where the object is detected. The similarity
measure that provides better results for this problem was the
correlation coefficient normalized. Because each object never
changes position, this template matching step is only done
once in the first captured image of the video sequence. Then,
these positions are used for all the images in the video
sequence.

b) *Finding the position of each person*

There has been much work in the area of people detection and
tracking using computer vision techniques. This problem has
been a big challenge since many real-time vision systems
require robustness in different environments. In the video
sequence of this project, two persons are talking with each
other and also moving randomly inside the scenario in order to
interact with these objects. The problems here are mainly:

1. The shape and size of each person can change over time because the persons can move far or close to the viewpoint in different parts of the scenario.
2. The persons can approach too much from each other in the video, making difficult the identification of each one.

3. One person can be partially occluded by the other person.

4. Some body parts of each person can be outside the scenario because of the viewpoint of the camera.

5. Each person moves in a random way.

In order to resolve this problem, a color-based tracking was used. Principally the color of the clothes of each person was used, assuming that both people in the video will have different color clothes. However, in order to make this people detection robust to the cluttered environment, a background subtraction technique and also blob detection was implemented in order to discriminate noise. The procedure can be described in this way:

1. Firstly, the first frame in the image was used in order to estimate the background image. This, of course, assumes that nobody is going to be present in the first frames of the video sequence. In order to detect the foreground, it was calculated the absolute difference of the pixel values between and image with the people present and the background image (using operator `cvAbsDiff` from OpenCV) [9]. Then, a threshold in the absolute difference image was applied. By this way, the foreground detected will be the silhouettes of each person. This technique, although is easy to implement and it has a very fast processing time, have the disadvantage of not discriminating the shadows of people.

2. Secondly, a Single-Gaussian Model [5] of each color was applied to the silhouettes in order to obtain the probability that the pixel values corresponds to a color. This Gaussian Model (mean and covariance matrix) was learned in an initialization step using samples manually taken of the clothes color of each person. The result of this process is a binary image where the pixels that are above a threshold are marked as belonging to each color.

3. After, a morphological opening operation was applied to the result image of the previous step. By applying this, we can obtain a more regular shape in the binary image. This was implemented using OpenCV operator `cvMorphologyEx` with a 3 x 3 square structuring element.

4. In this step, a blob detection and analysis was implemented in order to find the region of interest (ROI) that corresponds to each person. The analysis is made by using the size information of each blob in order to discriminate noise or false detections. After this analysis is done, we can estimate the bounding box of each person using the body human proportions.

5. In order to follow each person, the size of the bounding box of each person is increased for the next frame and the person is searched inside this area. This is a very simple tracking, however, it gives good results since the persons are not far from the camera and also they don't change position too fast.

This procedure, although, it works very well for all the video sequences of this project maybe it can not give the same result for other types of video sequence. Consequently, some

experiments were done using features tracking from each person, by this way, the idea is to identify and track using optical flow (pyramid-based KLT feature tracking) [15] a group of features conserving the distance properties between each feature of each group [14]. In these experiments some features were lost because of the occlusions, making it difficult to identify each person. However, this can be a very good approach for future research in order to improve the robustness of this system.

### c) Finding the motion direction of each person

In order to find the motion direction of each person, Motion Templates algorithms were used based on papers by Davis and Bobick [10] and Bradski and Davis [4]. These algorithms are very fast and robust. The implementation was done using OpenCV Motion Template functions. This functions can determine where a motion occurred, how it occurred, and in which direction it occurred. To calculate the motion direction of each person, the silhouettes (obtained by background subtraction as described in section 3.2) are updated in time using `cvUpdateMotionHistory` operator, after the motion gradient is calculated using the information of the temporal silhouettes (applying `cvCalcMotionGradient`), finally connected regions of Motion History pixels are found using OpenCV operator `cvSegmentMotion`.

With this result, we have region of motions with their gradient directions in the foreground image. In order to find what region of motion corresponds to each person, it was used the result of the procedure described in section 3.2. By this way, the gradient direction of the biggest region of motion inside the bounding box of each person corresponds to the motion direction of the person.

### 4) Human behavior analysis

Using the information described in the section 3, it's easy to know the human behavior at any time in the video sequence. In this step, we need to know principally:

1. If every object is near or far from each person.

2. The object toward which each person moves.

In order to find this information we only apply some rules and define thresholds using the result of the computer vision system (position of each fixed object and each person and motion direction of the persons). This human behavior information is sent by a socket TCP/IP connection to the multimodal high level data integration system in order to be fused with the speech recognition information (speech modality).

### 5) Results and conclusions

The computer vision system was executed in a laptop with an AMD Turion 64 2000 MHz processor, 1024 MB of RAM and a graphic card NVIDIA GForce Go 6150 (256MB). The computer vision system is capable of providing the required
information of all the 5 scenarios in real-time processing (25 frames by second). The system is robust to partial occlusions and it can detect and track both persons in any part of the scenario providing their position in all frames. In Figure 2, we can see the result of the computer vision system. The red circles with the red line inside is the motion direction (0 – 360 degrees) of each person, the blue pixels represent the Motion History of the human silhouettes, the white blob is the estimated area where each person is identified inside the scenario and the red small squares is the position of each fixed object (the computer, the books and the telephone).

In order to improve the robustness of the system in different illumination conditions, an approach of features tracking with optical flow can be used, by this way, the identification of each person is not going to rely only on the color and the size, also we're assuming that each person has different color clothes, this is not always true in a real scenario. Maybe for future research it can be used also particles filter approach in order to estimate the position of each person in the next frame improving the robustness of the tracking.

<figure><img src="image_15.png" /><figcaption>Figure 2: Video sequence of the first scenario.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 3: Results of the computer vision system</figcaption></figure>

### D. Ontology design and reasoning

In order to have a database for semantic reasoning, an ontology with comprehensive modeling pattern of multimodal actions and prior knowledge about the objects and user is necessary. We used open source tool Protégé [20] to create the ontology in classes, properties and individuals. Figure 4 is the looking of a developed ontology in Protégé 4.0 alpha environment.

<figure><img src="image_17.png" /><figcaption>Figure 4. Ontology in Protégé 4.0 alpha environment</figcaption></figure>

We use Soar [24] cognitive architecture in order to create and apply rules to query semantically the modality triples in the ontology. The ontology is re-implemented for adapting the specific working memory structure in Soar environment. To a certain range, class, property and individual in Protégé 4.0 alpha correspond to identifier, attribute and constant. Figure 5 shows an example of graph structure of the knowledge base. In order to have this knowledge base for reasoning, the ontology is created as states when the agent is initialized. After receiving the input from external application, an operator will be proposed and applied to map the input triple to the knowledge base then give out the result to the output link.
<figure><img src="image_18.png" /><figcaption>Figure 5. An example of graph structure</figcaption></figure>

For the interacting between Soar and external world, we use SML (Soar Markup Language) to send and receive commands packages as XML packets. The agent created by soar is loaded by the SML codes and run till there is any outcome or run for a certain number of circles which is set by codes too.

The outcomes are checked by SML code as well and kept for further actions. See Figure 6 for the execution cycle.

<figure><img src="image_19.png" /><figcaption>Figure 6. Execution cycle</figcaption></figure>

In our project, when we detect there is an input triple in the input-link, we will try to map it to ontology. Different operators will be proposed corresponding to the conditions. Then in the application stage of the operator, output will be generated and retrieved by the SML code. Further actions are taken according to the outputs.

## IV. EXPERIMENTS AND RESULTS

Let us consider one of the annotated example scenarios:

(1) [Beto] Hi Ronald! How is the life going?

(2) [Ronald] I am fine.

(3) [Ronald] I want to call Nick.

(4) [Beto] What for?

(5) [Ronald] He mentioned that he attended a wine tasting course.

(6) [Beto] It sounds interesting, I like wine.

(7) [Ronald] Actually I plan to join the next class. He also mentioned a book about French wines, but I cannot recall the name of the author.

(8) [Beto] Why don't you send a mail to Nick?

(9) [Ronald] Maybe I can find a book about it in the library.

(10) [Beto] Yes, you are right.

(11) [Beto] Did you find it?

(12) [Ronald] Yes, I did.

In our experimental work we used scenarios only with natural human language – we did not work with isolated words, commands, restricted language or something like that. I goal was to experiment with normal complete utterances expressed in a natural way. It should be noted here that we did not do spontaneous language like with pauses, interjections, hesitations, overlaps in speech, etc., because the wide-coverage CCG parser [8] that we used, is trained on newspaper texts, cannot analyze fragments of sentences, it accepts only well-formed complete utterances.

For us, to make multimodal data fusion means to interpret human behavior, to identify the users' plan, infer their intentions. Having understood what the people in the scene want to do, the system takes the decision about how to assist them in the given case. Looking at our challenging example scenario, we can see that there are 4 points in this dialog where the speakers express their plan to do something. In (3) Beto wants to call Nick, in (7) Ronald plans to attend the next class, in (8) there is a possible plan to send a mail and then this possible path of decision is changed for another route - in (9) there is an intention to find a book in the library. How shall the system realize what plans to take into account and what not? When to react and when not? This is why we employ multimodal information – when the plan is identified from the user's words, we look at the other modality data to see, if the person is going to “confirm” his words with the corresponding actions or not. In (3), (7) and (8) the persons expressing the said intentions were standing still, just continuing the talk. And only in the (9) utterance Ronald moved to the book-shelves. That is why only in this last case of plan expression the system reacted and prompted where to find the desired book. By the way, in the phrase “Maybe I can find a book about it in the library” we have to resolve ambiguity between the library in the room and a library on the web. We again do that using information from the other modality – we look if the person is moving to the books in the room or if he is moving to the computer.

Schematic examples of decision making process:

(3) If (Ronald) [wants to send] {email to Nick} &
(Ronald [is moving to] {the computer} | He
[is close to] {the computer}) then

>> open the mail client with the "to" field filled
with nick@uclouvain.be

(9) If (Ronald) [can] find {book} [about] {it} [in]
{the library} &

(Ronald [is moving to] {the library} then

>> There is a book about French wines on the
first shelf.

(9) If (Ronald) [can] find {book} [about] {it} [in]
{the library} &

(Ronald [is moving to] {the computer})
then

>> Open a web search website and put the
keyword in the search field

When identifying the person's plan from speech, we basically rely on the linguistic semantic analysis as described in Section III B, but we certainly take advantage of the obvious lexical
signs of plan and intentions expression. For example, such
verbs and phrases like “want”, “wish”, “plan”, “going to”, etc.
(we defined 19 expressions like this in total) in the certain
syntactic context and in the present or future tense clearly
point at the person’s intention to do something. And vice
versa, negative forms of verbs like “I don’t want”, “I have no
wish to…”, “You don’t want to…” as well as verbs in the past
tense serve as stop-words, and signal that this plan should be
discarded and not taken into account, because no system
response is needed.

V. CONCLUSIONS AND FUTURE WORK

During this challenging project we have posed and solved
several difficult problems. We have:

* managed to deal with spatial relationships (based on the fixed “anchor” objects in the room)

* made semantic fusion of events not coinciding in time

* achieved good results in speaker identification - synchronisation between image and speech identification

* created an open framework to manage fusion between two (in our case) or more modalities (in enhanced future work)

* designed the system so that each component can run in a separated machine thanks to the distribution mechanism interchanging data through a TCP/IP network.

However, we are not going to stop at this point, but we
have even more problems to solve in our future work. To
name just a few, we should:

* implement effective learning mechanism

* perform efficient decision making even from information fragments

* handle spatial relationships relatively to moving people

* perform 3D video analysis

* identify detection of orientation of the people in the scene

* add at least one more modality - eye gaze tracking

* recognize various types of gestures

* learn to deal with natural language redundancy (repeating the same idea in different words).

VI. ACKNOWLEDGEMENT

The research described herein was started under the European
FP6 SIMILAR Network of Excellence (www.similar.cc) and
continues by the European FP6-35182 OpenInterface project
(www oi-project org).

We thank Diego Ruiz and Ronald Moncarey (UCL-TELE,
Belgium) for precious help in the project preparation.

VII. REFERENCES

[1] A Gentle Introduction to Soar: 2006 update, http://ai.eecs.umich.edu/soar/sitemaker/docs/misc/GentleIntroduction-2006.pdf

[2] Bolt R. A., "Put-that-there: Voice and gesture at the graphics interface," in *International Conference on Computer Graphics and Interactive Techniques*, July 1980, pp. 262–270

[3] Bos J. et al., “Wide-coverage semantic representations from a CCG-parser” Proc. of Int. Conf. COLING, 2004

[4] Bradski G., Davis J. "Motion Segmentation and Pose Recognition with Motion History Gradients" IEEE WACV'00, 2000

[5] Caetano T.S., Olabarriaga S.D., Barone B.A.C., "Performance evaluation of single and multiple-Gaussian models for skin color modeling", in: Proc. XV Brazilian Symposium on Computer Graphics and Image Processing, 2002, pp. 275-282

[6] Chai J., Pan S. and Zhou M., MIND: A Context-based Multimodal Interpretation Framework, Kluwer Academic Publishers, 2005

[7] Cole L., Austin D., Cole L., "Visual Object Recognition using Template Matching", Proceedings of Australasian Conference on Robotics and Automation, 2004

[8] Curran J., Clark S. and Bos J. (2007): *Linguistically Motivated Large-Scale NLP with C&C and Boxer*. Proceedings of the ACL 2007 Demonstrations Session (ACL-07 demo), pp.29-32.

[9] Davis J., Bobick A., “A Robust Human-Silhouette Extraction Technique for Interactive Virtual Environments”, IFIP Workshop on Modeling and Motion Capture Techniques for Virtual Environments (CAPTECH98), November 1998

[10] Davis J., Bobick A., “The Representation and Recognition of Action *Using Temporal Templates*” MIT Media Lab Technical Report 402, 1997

[11] Heijmans H.J.A.M., "Morphological image operators" Advances in Electronics and Electron Physics, P. Hawkes (Ed.), Ac. Press: Boston, suppl. 24, Vol. 50, 1994

[12] Intel Corporation, Open Source Computer Vision Library – OpenCV, http://www.intel.com/technology/computing/ opencv/index.htm, 2008

[13] Kamp H. and Reyle U., "From Discourse to Logic. Introduction to Model-theoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory", Kluwer, Dordrecht, 1993.

[14] Kolsch M., Turk M. “Fast 2D Hand Tracking with Flocks of Features and Multi-Cue Integration” Proceeding of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop (CVPR’04)

[15] Lucas B.D., Kanade T., “An Iterative Image Registration Technique with an Application to Stereo Vision” In Proc. Imaging Understanding Workshop, pages 121–130, 1981

[16] Oviatt S. et al., “Toward a theory of organized multimodal integration patterns during human-computer interaction”, Proc. of the Int. Conf. ICMI, 2003

[17] Oviatt S. and Cohen P., “Multimodal interface research: A science without borders,” *Proc. of 6th Int. Conference on Spoken Language Processing*, vol. 43, no. 3, pp. 45–53, 2000

[18] Pfleger N., “Context based multimodal fusion”, Proc. of the Int. Conf. ICMI, 2004

[19] Pfleger N. and Alexandersson J. "Towards Resolving Referring Expressions by Implicitly Activated Referents in Practical Dialogue Systems" In: Proc. of the Workshop on the Semantics and Pragmatics of Dialogue, 2006.

[20] Protégé http://protege.stanford.edu/

[21] Reynolds D., Rose R. Robust text-independent speaker identification using Gaussian mixture speaker models // Speech and Audio Processing, IEEE Transactions on, Vol. 3, No. 1. (1995), pp. 72-83

[22] Ruiz D. and Macq B., A master-slaves volumetric framework for 3D reconstruction from images, // Proc. SPIE, Volume 6491. Videometrics IX, J.-Angelo Beraldin, Fabio Remondino, Mark R. Shortis, Editors, 64910G (San Jose, CA, USA, 2007)
[23] Soar: A Functional Approach to General Intelligence, http://ai.eecs.umich.edu/soar/sitemaker/docs/misc/SoarFunction.alOverview.pdf

[24] Soar http://sitemaker.umich.edu/soar/home

[25] Vybornova O., Gaitanis K. and Macq B., "Plan recognition using multimodal integration," 2006, Proc. of Int. Conf. CogSci, 2006.

[26] Vybornova O., Gemo M., Macq B.: Multimodal Multi-Level Fusion Using Contextual Information // ERCIM News, No. 70, July 2007.

[27] Vybornova O., Gemo M., Moncarey R., Macq B.: Ontology-Based Multimodal High Level Fusion Involving Natural Language Analysis for Aged People Home Care Application // In: Proceedings of INTERSPEECH 2007, August 27-31, Antwerpen, Belgium.

[28] Vybornova O., Gemo M., Moncarey R., Macq B.: Contextual Information Sharing in Natural Language and Gesture Crossmodal Integration for Aged People Assistive Home Care Application // In: Proceedings of the 29th Annual Conference of the Cognitive Science Society (CogSci 2007), August 1-4, Nashville, USA.

[29] W. Walker, et. al. "Sphinx-4: A Flexible Open Source Framework for Speech Recognition" Technical Report, SMLI TR2004-0811, 2004 SUN MICROSYSTEMS INC.

several companies (Real Time Services, Infomedia, Banco Azteca, Cognosite) learning and using different software programming technologies. In 2007, he obtained a scholarship from the Mexican Government to study a PhD in Computer Vision at Télécom SudParis (Evry, France). He is currently studying his second year of his PhD with the thesis subject ?3D Human Gesture Acquisition by Computer Vision and Virtual Rendering?. He has big research interest in Computer Vision, Artificial Intelligence and Computer Graphics. Other interests include videogames, learning new languages, eating pizza, listening music and meeting people.

Ao Shen has a bachelor's degree in Communication Science & Engineering at Fudan University, Shanghai, China and a bachelor's degree in Electronic Communication Engineering at Birmingham University, UK. He is experienced in coding mobile application and won the second prize in Vodafone Betavine Student Competition. Now he has finished his first year of PhD study in Electronic Electrical & Computer Engineering Department, University of Birmingham. His main interests multimodal systems and semantic level applications, knowledge base and ontology creation.

**Olga Vybornova** received a PhD degree in applied and mathematical linguistics from Moscow State Linguistic University, Russia, in October, 2002. From May 2005 she was a post-doctoral researcher and from January 2008 to the present time a research assistant in Université Catholique de Louvain at the Communications and Remote Sensing Lab (UCL-TELE) in Louvain-la-Neuve, Belgium.

Her main research interests are in the fields of multimodal data integration, assistive technology supporting active ageing and social cohesion, semantics-driven multimedia presentation generation, context-based inference, meaning representation, goals, intentions and commitments in communication, dialog modeling; cognitive linguistics; memory and attention, computational semantics.

**Hildeberto Mendonça** has a bachelor's degree in Computer Science at University of Fortaleza, Brazil. During his undergraduate studies he worked in the Laboratory of Human Computer Interaction with distance learning and digital TV projects. Afterwards, he worked in the Laboratory of Knowledge Management with public security and knowledge bases projects. Currently, he has finished his Diploma of Extended Studies in Human Computer Interaction Meta Representation as a requirement for the PhD in Applied Science in the field of Multimodal User Interaction at Catholic University of Louvain, Belgium. He has a rich professional experience in web application development, collaborative systems, and software engineering and software architecture projects. His technical skills are in the Java platform and its main extensions, C++, XML, relational databases, distributed systems, web services, web and desktop user interfaces, and creation and maintenance of ontology.

**Daniel Neiberg** received an Master of Science in field of electrical engineering, KTH (Royal Institute of Technology), Stockholm, Sweden, in 2003. From 2002 to 2006 he worked as a Research engineer at Centre of Speech Technology (CTT) at KTH, were he was doing work in Speaker Verification and Emotion Recognition. From 2006 - 2007 he attended a Trainee Internship at Asahi Kasei, where the working tasks included development and evaluation of Automatic Speech Recognition of Swedish. In 2007, he enrolled as PhD student at Centre of Speech Technology (CTT) at KTH, where the main field of research is methods for automatic speech recognition and speech production.

**David Antonio Gómez Jáuregui** was born in Mexico City on June 13, 1980. He received a BSc in Computer Systems Engineering in 2002 and a MSc. in Computer Science in 2004 with a specialization in Intelligent Systems and Computer Vision. He obtained these grades from Tec de Monterrey Campus Cuernavaca, Mexico. In 2003 he participated in the Robotic Soccer tournament (FIRA 2003) celebrated in Vienna, Austria in representation of Tec de Monterrey. During 2004 - 2007 he worked as a software developer in
# Sign-language-enabled information kiosk

Pavel Campr, Marek Hrúz, Alexey Karpov, Pınar Santemiz, Miloš Železný, and Oya Aran

***Abstract***—This paper presents design and creation of multimodal sign-language-enabled information kiosk which was developed during eNTERFACE’08 workshop. The kiosk uses automatic computer-vision-based sign language (SL) recognition, automatic speech recognition (ASR) and touchscreen as input modalities. The outputs are presented on a screen displaying 3D signing avatar and on a touchscreen showing special graphical user interface designed for the deaf users. The kiosk was tested on a dialogue providing information about train connections, but the scenario can be easily changed. The kiosk can be used both by hearing (ASR can be used) and deaf users (even who cannot read), in several languages. The human-computer interaction is controlled by a computer-driven dialogue system. A prototype of the kiosk was built and the work experience raised some new usability questions.

***Index Terms***—multimodal information kiosk, sign language recognition, sign language synthesis, dialogue system, automatic speech recognition

## I. INTRODUCTION

THE aim of this eNTERFACE 2008 project is to design an information kiosk for deaf people that will use sign language (SL) as a main communication mean. Background of the idea is based on current research on sign language processing carried out at UWB (University of West Bohemia) and BU (Boğaziçi University). Methods for synthesis (visual animation) and recognition of sign language are under research. The project also follows the work done during previous eNTERFACE workshops (project No. 3 *Sign Language Tutoring Tool* at eNTERFACE 2006 in Dubrovnik [1] and project No. 3 *A Multimodal Framework for the Communication of Disabled* at eNTERFACE 2007 in Istanbul).

Deaf or hearing-impaired users have limited possibility of communication with hearing people, which can be a problem especially in the case of communication with authorities or information providers (train connection, sales, etc.) These people also cannot use speech-based automatic information services. In these cases dialogue systems should be designed to be accessible by deaf users to solve this problem. This project aims to design a dialogue system (information kiosk) in this way.

The idea is to combine sign language recognition and synthesis tasks to develop a simple information kiosk [2] for providing information such as train connections for deaf people who use sign language. The kiosk will be a standalone

Pavel Campr, Marek Hrúz and Miloš Železný are with the Department of Cybernetics, Faculty of Applied Sciences, University of West Bohemia, Pilsen, Czech Republic.

Oya Aran and Pınar Santemiz are with the Computer Engineering Department, Boğaziçi University, İstanbul, Turkey.

Alexey Karpov is with the Speech and Multimodal Interfaces Laboratory, St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, St. Petersburg, Russia.

machine equipped with a standard PC, a touch screen display, a big screen for an avatar a microphone, and one or several cameras. The cameras will capture body/facial gestures of a signing person, while the big screen will present the rendered sign language output and the touch screen will allow touch commands as an alternative input. The system (kiosk) will wait for an input from user. When the user occurs in front of the kiosk, it will start its SL recognizer. It will decode the information user is looking for, detect important signs for obtaining data needed (time, destination, etc.) and requests from the user. As an output, information based on this data will be produced.

## II. SYSTEM OVERVIEW

### A. The information kiosk setup

Similarly as in case of speech-based information systems, we need to design a concept for sign language-based information system. For such a system special hardware will be required to be able to acquire input data. To be able to recognize sign language (both manual gestures and oral articulation) we need to collect both detailed (facial) and upper body visual data. It is putting restrictions to the recording conditions. We present the design of hardware setup for such information system as an information kiosk allowing to be used in public places, such as train station and at the same time allowing to capture data allowing best possible recognition rates.

The information system can be divided into several basic parts: central control unit, input sensors, and output part. Input sensors comprise a visual part, an acoustic part, and a touch screen part. Output is presented to the user using graphical

<figure><img src="image_20.png" /><figcaption>Fig. 1. Setup of cameras proposed for information kiosk for deaf users. This setup was used for recording SLR-A and SLR-B databases.</figcaption></figure>
<figure><img src="image_21.png" /><figcaption>Fig. 2. Modules overview</figcaption></figure>

screen. Central control unit will be typically a multimedia PC equipped with special hardware for recording and processing of the input data. Central control unit will also take care of the connection to external data sources, such as various types of databases (about train connections etc.)

Visual input sensors are made up by cameras situated in the rear part of the kiosk. Camera 1 looks from horizontal view at the user (the upper part of the body) standing in front of the kiosk. Camera 2 is situated at the top of the rear part of the kiosk. It looks downwards at the user's whole body. It enables to gather 3D information about the position of hands. Camera 3 is situated near Camera 1, but looks at the user's face. Data from Camera 3 will be used for automatic lip-reading, facial expressions and head gestures. Whole hardware setup is depicted in Figure 1.

Acoustic input sensor will be a microphone situated at the top of the screen, closest to the user. It should capture voice of the user with highest possible sensitivity. Acoustic part is an optional part of the system and can be used in case when communication of hearing (and speaking) people with the system is expected. However in this case, the expected recognition rate of such speech will be very low and can be used only as an accompaniment.

Haptic input sensor will be realized as a touch screen. For some easy tasks such as selection from several choices it will be more efficient to present graphically several choices and let user select by touching the screen at the position of one of the choices instead of forcing him to use sign language, as the expression in sign language can be complicated (i.e. city names that has to be spelled etc.)

Graphical screen output will be used to guide the user through the whole information providing process, allowing him/her to quickly enter most frequently used modes, letting him to choose from several choices when appropriate, and giving him feedback for his/her actions. Design of the graphical user interface will be done with respect to the special needs of users and will be as intuitive as possible to efficiently help the user to reach his/her goal.

### III. DIALOGUE CONTROL MODULE

#### A. Computer-driven dialogue

The interaction between the user and the kiosk is managed by a computer-driven dialogue system (Fig. 3). The dialogue consists of several scenarios, which are defined as a set of questions, their possible answers and answer generator. When

<figure><img src="image_22.png" /><figcaption>Fig. 3. Dialogue flow</figcaption></figure>

the user selects a scenario (e.g. train connection information), the dialogue control asks the user for an answer of the first unanswered question. The answer can be entered as one- or multiple-word sentence (supported by speech recognizer, planned for SL recognizer in the future). All entered sentences must satisfy a context-free grammar defined by Backus-Naur Form (BNF). The grammar allows to generate a list of all sentences, their parts or words. All possible words, which can follow in the current sentence, are displayed in GUI (fig. 4, right). Later it allows to validate the user's answer. When all required questions are answered the dialogue system generates an answer for the user's query.

Scenarios are defined in one configuration file (YAML format) which can be easily modified. Here is an example with a part which define "departure information" scenario:

```
dialogue:
  ...
  screens:
    layout:
      template: layout.html
    ...
    departures:
      template: body departures.html
      prompt: here you can find information about
              departures from this train station
      legend: departures
      form:
        departure:
          prompt: select departure
          label: departure
          type: grammar
          grammar: towns
          default: "town_prague."
        destination:
          prompt: select destination
          label: destination
          type: grammar
          grammar: towns
        date:
          prompt: select date of departure
          label: date of departure
          type: grammar
          grammar: date
        ...
```

The default language used in the system is English. Other languages are translated using internationalization configuration file which contains all necessary translations from English to other languages (Czech, Turkish, Russian, etc.) both spoken and signed (for sign synthesis).
provided the information we have is the mean, variance and gain of each Gaussian in the mixture. We have precomputed the likelihoods and stored them in a look-up table. The range of the likelihood is from 0 to 255. A likelihood of greater than or equal to 128 means that the particular color belongs to a skin segment. With this procedure we obtain a 256x256x256 look-up table containing the likelihood of all the colors from RGB color space. The segmentation is straightforward. We obtain the likelihood of the color of each pixel from the look-up table. According to the likelihood of the color, we decide whether it belongs to a skin segment or not. For each frame, we create a mask of skin color segments by thresholding the likelihood image. The result is as shown in Fig 6.

<figure><img src="image_23.png" /><figcaption>Fig. 8. (a) Original image, (b) Particle distribution with joint PF, (c) Estimated hand positions</figcaption></figure>

### C. Tracking

1) **The Joint Particle Filter (PF):** We use a joint PF that calculates a combined likelihood for all objects by modeling the likelihood of each object with respect to others. Figure 7 shows the pseudo-code of the joint PF. $x_t^n$ is the joint object state, as defined in Eq.1. The first step is to determine the initial states and the weights of the particles, with respect to the prior distribution. The initial distribution of the particles are determined with respect to an explicit detection step based on connected component labeling. The particles at time $t$ are determined by the re-sampling, prediction and weight setting steps. At the re-sampling step, new particles are sampled with replacement from the weighted particles at time $t-1$. At this step the weights of the new particles are equally assigned. The states of the re-sampled particles at time $t$ are determined by the object dynamics and by an additional mean-shift step. The weights are determined by normalizing the joint likelihood.

<figure><img src="image_24.png" /><figcaption>Fig. 7. Joint PF algorithm</figcaption></figure>

2) **Object Description:** The state vector for a single object consists of the position, the velocity and the shape parameters. The shape parameters are selected as the width, the height and the angle of an ellipse surrounding the object. Thus, for a single object we have a seven dimensional state vector. Then, the joint particle is a single vector containing all the objects in the scene:

$$ \mathbf{x}_t^n = \{\mathbf{x}_t^{n,f}, \mathbf{x}_t^{n,r}, \mathbf{x}_t^{n,l}\}^T \quad (1) $$

where $f, r, l$ are indexes to the face, right and left hands. In the rest of the paper, we will refer to $\mathbf{x}_t^n$ as the joint particle and $\mathbf{x}_t^{n,i}$ as the particle or as the sub-particle, alternatively.

<figure><img src="image_25.png" /><figcaption>Fig. 9. (a) The thresholded image that is used in likelihood calculation, (b) hand region and different particles. The likelihood function gives the highest likelihood for the particle at the top.</figcaption></figure>

3) **Dynamic Model:** For each object, the position and the velocity parameters are modeled by a damped velocity model and the shape parameters are modeled by a random walk model.

For multiple objects in the joint PF, the dynamic model is applied to each object. We additionally apply mean shift [4] to the sub-particles of each object independently. The MS algorithm moves the particle centers to the areas with high skin color probability. This allows us to use particles effectively, since the particles with low weights will be less likely. As a result, a PF with MS needs fewer particles than a standard PF.

4) **Appearance Model:** We use the skin color probability image, which has a positive probability for skin color pixels and zero probability for other colors.

To calculate the likelihood of a single object, we make two measurements based on the ellipse that is defined by the state vector of the particle:

* **A:** The ratio of the skin color pixels to the total number of pixels inside the ellipse.

* **B:** The ratio of the skin color pixels to the total number of pixels at the ellipse boundary.

These two ratios are considered jointly in order to make sure that our measurement function gives high likelihood to particles that contains the whole hand without containing many non-hand pixels [5]. If we do not take the ellipse boundary into account, smaller ellipses are favored and particles tend to get smaller. We design our measurement function Eq.2 to return
a high likelihood when A is as high as possible and B is as low as possible:

$$z_t^{n,i} = \begin{cases} 0 & , \text{ if } A < \Phi_p \\ 0.5 \cdot A + 0.5 \cdot (1 - B) & , \text{ otherwise} \end{cases} \quad (2)$$

where $z_t^{n,i}$ denotes the likelihood of a single object, i, for the particle n.

The first line in Eq.2 is required to assign low likelihood to particles that have zero or very few skin color pixels. Otherwise these particles will receive 0.5 likelihood value even if they do not contain any skin color pixels. The equation takes its highest value when there are no skin colored pixels at the boundary ($B = 0$), and when all the inner pixels are skin colored. Figure 9b shows the grey-level hand image and possible particles. The particle at the top receives the highest likelihood by Eq.2 where as the other particles receive lower likelihoods.

5) Joint Likelihood Calculation: A joint particle is a combination of sub-particles which refer to the objects we want to track, i.e. two hands and the face (Eq.1). We calculate the joint likelihood with respect to the following:

1) The likelihood of a single object based on the appearance model;

2) The distance of each object to the other objects to handle interactions. Assign low likelihood if the sub-particles are close to each other;

3) Additional constraints on respective object locations. This criterion is needed to prevent wrong object assignments especially after occlusions.

We define partial likelihoods for each criterion above and calculate the joint likelihood by the multiplication of the partial likelihoods.

The joint likelihood is calculated for the objects that stay in the scene. If any of the objects disappears, it is excluded from the likelihood calculations. We assume an object has disappeared if all of the sub-particles of that object have zero weight.

## D. Handshape Feature Extraction

In order to recognize the signs we also need shape information in addition to the tracking features. In order to describe the shape we implemented five algorithms and tested their performances over the set of finger alphabet in Czech Sign Language, which can be seen in Fig 10.

After segmenting and tracking of the hands, we obtain the segmented hand image for each sign. Using this image, we find the contour and the gray scale image of the hand. Then we find DCT coefficients from the gray scale image [6], Hu moments from the segmented hand image [7], and Fourier descriptors from the contour points of the hand shape [8]. The feature extraction methods can be seen in Fig 11.

From the points on the contour, we get the following:

* Complex coordinates

* Distances of the points from the centroid of the hand

* Angles between two consecutive points

<figure><img src="image_26.png" /><figcaption>Fig. 10. Finger alphabet of the Czech sign language</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>Fig. 11. Feature extraction methods</figcaption></figure>

Then we calculate Fourier descriptors using these features. The procedure of obtaining Fourier descriptor features can be seen in Fig 12.

In the first method, to eliminate the effect of translation, we first subtract from each coordinate the center, compute the Fourier transform of the complex coordinate and find its absolute value. So, if the point is $P_i = (x_i, y_i)$ and the center of the hand is $P_c = (x_c, y_c)$, then the first feature is

$$f_i^1 = \text{abs} \left[ \text{fft} \left( (x_i - x_c) + i(y_i - y_c) \right) \right] \quad (3)$$

In the second method, we find the distance of the boundary points from the centroid of the shape. So,

$$f_i^2 = \text{abs} \left[ \text{fft} \left( \sqrt{(x_i - x_c)^2 + (y_i - y_c)^2} \right) \right] \quad (4)$$

<figure><img src="image_28.png" /><figcaption>Fig. 12. The procedure to obtain the complex coordinates, central distances and the angle between consecutive points</figcaption></figure>
In the third method, we first compute the angles between two consecutive points and compute the fft features using the formula

$$f_i^3 = \text{abs} \left[ \text{fft} \left( \arctan \frac{y_{i+1} - y_i}{x_{i+1} - x_i} \right) \right] \quad (5)$$

In the fourth method, we compute the DCT features from the gray scale image of the hand [6]. Then we trace the upper-left corner of the DCT image diagonally and choose the first 15 features and eliminate the first feature, which is the DC component.

Finally in the last method, we compute the Hu moments from the segmented hand image [7]. In order to find the Hu moments, we first compute the internal moments of order p+q using the formula

$$m_{pq} = \iint (x - x_c)^p (y - y_c)^q dxdy \quad (6)$$

where x, y are the pixel coordinates belonging to the hand. Then we normalize the moments for eliminating the scale factor

$$n_{pq} = \frac{m_{pq}}{m_{00}^{p+q+1}} \quad (7)$$

We use seven Hu moments which are invariant to rotation. The first six are also reflection invariant whereas the seventh moment is skew orthogonal invariant, which is useful in distinguishing mirror images.

$$ 
\begin{aligned}
S_1 &= n_{20} + n_{02} \\
S_2 &= (n_{20} + n_{02})^2 + 4n_{11}^2 \\
S_3 &= (n_{30} - 3n_{12})^2 + (n_{03} - 3n_{21})^2 \\
S_4 &= (n_{30} + n_{12})^2 + (n_{03} + n_{21})^2 \\
S_5 &= (n_{30} - 3n_{12}) \cdot (n_{30} + n_{12}) \cdot \\
    & \quad ((n_{30} + n_{12})^2 - 3(n_{03} + n_{21})^2) \\
    & \quad -(n_{03} - 3n_{21}) \cdot (n_{03} + n_{21}) \\
    & \quad \cdot (3(n_{30} + n_{12})^2 - (n_{03} + n_{21})^2) \\
S_6 &= (n_{20} - n_{02}) \cdot ((n_{30} + n_{12})^2 - (n_{03} + n_{21})^2) \\
    & \quad + 4n_{11}^2 \cdot (n_{30} + n_{12}) \cdot (n_{03} + n_{21}) \\
S_7 &= (3n_{21} - n_{03}) \cdot (n_{30} + n_{12}) \cdot \\
    & \quad ((n_{30} + n_{12})^2 - 3(n_{03} + n_{21})^2) \\
    & \quad -(n_{03} - 3n_{12}) \cdot (n_{03} + n_{21}) \\
    & \quad \cdot (3(n_{30} + n_{12})^2 - (n_{03} + n_{21})^2)
\end{aligned}
$$

In our experiment, we use a database containing 241 images from the Czech sign language finger alphabet. In this database there are 23 classes which are some of the letters in the alphabet. Each class contains six to 18 samples. To create our training set, we select five samples from each class randomly and put the remaining samples to the test set. We form six folds in total, where we guarantee that each sample is put at least once to the training set. So, in each fold training set includes 115 samples and test set includes 126 samples. We use 1-nearest neighbor method as a classifier. The recognition results can be seen in Table V-D.

According to our results, DCT obtained significantly better results than the others. The reason for that is DCT also takes

<table><caption>TABLE I
RECOGNITION RESULTS USING THE SET OF CZECH SIGN LANGUAGE FINGER ALPHABET</caption><thead><tr><th>Method</th><th>Accuracy</th><th>Std Deviation</th></tr></thead><tbody><tr><td>FFT-Complex</td><td>22%</td><td>2,05%</td></tr><tr><td>FFT-Centroid</td><td>23%</td><td>2,51%</td></tr><tr><td>FFT-Angle</td><td>17%</td><td>3,70%</td></tr><tr><td><b>DCT</b></td><td><b>75%</b></td><td><b>3,92%</b></td></tr><tr><td>Hu Moments</td><td>30%</td><td>2,54%</td></tr></tbody></table>

<figure><img src="image_29.png" /><figcaption>Fig. 13. Examples of images that are misclassified by the DCT method</figcaption></figure>

into account the gray scale texture information in the hand region whereas the others only use shape information.

Some signs have a similar shape, so using only shape information it is difficult to differentiate between them. Some examples can be seen in Fig 13.

## E. Sign Recognition

For the purpose of recognition we use Hidden Markov Model (HMM). The signs are modeled as an 8-state HMM (two of these states are non-emitting). Each state is modeled with a single Gaussian. This was due to the relative low amount of data. The structure of the HMM can be seen in Fig. 14.

One model of a sign was trained on 4 out of 6 video sequences. PCA and ICA were tested to reduce the dimensionality of the data and to align the data according to the feature space coordinate system. Both methods failed to improve the recognition rate since there were too few samples available. The remaining two video sequences were used to test the system. Results can be seen in Table II.

<figure><img src="image_30.png" /><figcaption>Fig. 14. The structure of HMM used for modeling the signs.</figcaption></figure>
<table><caption>TABLE II
RECOGNITION RATE OF SLR</caption><thead><tr><th>No. parameters</th><th>Method</th><th>Recognition rate</th></tr></thead><tbody><tr><td>49</td><td>x</td><td>81.63%</td></tr><tr><td>49</td><td>PCA</td><td>78.57%</td></tr><tr><td>39</td><td>PCA</td><td>78.57%</td></tr><tr><td>29</td><td>PCA</td><td>76.53%</td></tr><tr><td>19</td><td>PCA</td><td>70.41%</td></tr><tr><td>9</td><td>PCA</td><td>47.96%</td></tr></tbody></table>

The hands of a signer were approximated as ellipses. Following features were extracted :

* Center of the ellipse

* Size of the ellipse (width and height)

* Angle between the image x-axis and the major axis of the ellipse

* Velocity of the ellipse

* DCT coefficients of images of both hands

Out of 338 video sequences 15 were not tracked correctly. DCT coefficients improved the recognition rate significantly.

## VI. SIGN LANGUAGE SYNTHESIS MODULE

### A. Feedback Animation

1) **Animation Model:** The feedback animation module can be divided to a module for rendering the animation model and to a module for forming the animation trajectories (a trajectory generator). 3D geometric animation model of the avatar is in compliance with the H-Anim standard. Currently the animation model covers 38 joints and body segments. Each segment is represented as textured triangular surface, 16 segments is used for fingers and the palm, one for the arm and one for the forearm, totally 1372 vertices and 1772 triangles per hand. The thorax and the stomach are together represented by one segment, 182 vertices and 360 triangles. The talking head is composed from seven segments, totally 4692 vertices and 9243 triangles.

The body segments are connected by the avatar skeleton. One joint per segment is sufficient for this purpose. A controlling of the skeleton is carried out thought the rotation of segments (3 DOF per joint). The rotation of the shoulder, elbow, and wrist joints are completed from 3D positions of the wrist joints by the inverse kinematics.

The animation of the talking head is performed by a local deformation of the triangular surfaces. It is primarily used for the animation of the avatar's face and the tongue [9]. The triangular surfaces are deformed according to influence zones defined on the triangular surface by spline functions constructed from several 3D control points. The collection of these points is currently taken by 9 animation parameters. The rendering of the animation model is implemented in C++ and OpenGL code.

2) **Trajectory Generator:** Firstly for the manual component of signed speech, the trajectory generator performs the syntactic analysis of the symbolic string on the input HamNoSys string and creates parse tree structure. Currently the trajectory generator uses 374 parse rules. However, it is difficult to define rules and actions for all symbol combinations to cover

<figure><img src="image_31.png" /><figcaption>Fig. 15. Left panel: The list of all items. Right panel: An example of the items stored in the definition file.</figcaption></figure>

<table><caption>Fig. 16. The statistic of symbols, rules, and actions used by the HamNoSys parser.</caption><thead><tr><th rowspan="2">Block</th><th colspan="2">HamNoSys Symbols</th><th colspan="2">Parser</th></tr><tr><th>#Base</th><th>#Auxiliary</th><th>#Rules</th><th>#Actions</th></tr></thead><tbody><tr><td>Symmetry</td><td>4</td><td>0</td><td>8</td><td>8</td></tr><tr><td>Handshape</td><td>12</td><td>11</td><td>35</td><td>6</td></tr><tr><td>Finger and palm orientation</td><td>26</td><td>3</td><td>34</td><td>5</td></tr><tr><td>Location</td><td>30</td><td>17</td><td>85</td><td>14</td></tr><tr><td>Action</td><td>40</td><td>42</td><td>204</td><td>15</td></tr><tr><td>Link of blocks</td><td>0</td><td>0</td><td>8</td><td>1</td></tr></tbody></table>

the entire notation variability. We made a few restrictions in order to preserve maximum degree of freedom. In this assumption, the annotation of the sign have a good meaning for the user familiar with HamNoSys as well as signs are obvious enough for the transformation to the avatar animation. Next, the structurally correct string is decomposed to nodes determined by parsing rules. Two key frames data structure to distinguish the dominant and non dominant hand are used. The data structure of the key frame is composed from items specially designed for trajectory generation purpose (Figure 15 on the left). Next step of trajectory forming is filling of the leaf nodes from the symbol descriptors stored in the definition file (Figure 15 on the right). The definition file covers 138 HamNoSys symbols.

39 rule actions were added in a manner that one rule action is connected with each parse rule. The parse tree is processed by several tree walks. The initial tree walk put together the items of the key frames according to the type of the rule actions. The reduced tree is joined and transformed to the trajectories accordance with the timing of the particular nodes. Finally, the final trajectories for both hands are contained in the root node. The number of used symbols, parse rules, and actions used in the system is summarized in Figure 16. The final step is the conversion of the trajectories into the avatar animation and synchronization with articulatory trajectories generated by the selection of articulatory targets [9].

3) **Evaluation of Synthesized Signs by Deaf:** A subject evaluation of quality of synthesized signs has been performed in advance with the manual and non-manual components of a sign. The perception test by deaf children from primary school
<table><caption>TABLE III
SIGN LANGUAGE SYNTHESIS UNDERSTANDING EXPERIMENTS</caption><thead><tr><th>Experiment</th><th>1</th><th>2</th></tr></thead><tbody><tr><td>Test 1</td><td>80%</td><td>95%</td></tr><tr><td>Test 2</td><td>85%</td><td>80%</td></tr></tbody></table>

## VII. AUTOMATIC SPEECH RECOGNITION MODULE

A speaker-dependent automatic speech recognition (ASR) system was developed and embedded into the information kiosk as an optional alternative to the interactive GUI and the interface based on sign language recognition. ASR system is multilingual one and able to recognize voice commands both in English and Czech. The lexicon of ASR contains 101 diverse words for each language (Czech towns, digits, names of months and weekdays, etc.)

### A. Signal processing and feature extraction

The audio signal is captured by a microphone of a headset and sampled at 16 KHz with 16 bits on each sample using a linear scale. The system is intended for the distant talking and human-computer interaction, so the microphone was selected to be able to capture speech signal at the distance of 2 meters from a speaker with an acceptable SNR. The signal is divided into the frames and cepstral coefficients are computed for the 25 ms overlapping frames with 10 ms shift between adjacent frames applying the bank of triangular filters calculated according to the Mel-scale frequencies 8:

$$ Mel(f) = 2595 \log_{10} 1 + \frac{f}{700} \quad (8) $$

Mel-frequency cepstral coefficients (MFCCs) are calculated from the log filterbank amplitudes using the discrete cosine transform. So the audio speech recognizer system calculates 13 MFCCs (including 0-th coefficient) as well as estimates the first and second order derivatives that forms an observation vector of 39 components. The acoustical modeling is based on left-right continuous Hidden Markov Models (HMMs) [10], applying mixtures of Gaussian probability density functions that are defined according to the equation 9:

$$ \mathcal{N}(o; \mu, \Sigma) = \frac{1}{\sqrt{(2\pi)^n |\Sigma|}} e^{-\frac{1}{2}(o-\mu)^T \Sigma^{-1} (o-\mu)} \quad (9) $$

where $\mathcal{N}$ is a Gaussian with mean vector $\mu$ and covariance matrix $\Sigma$, and $n$ is the dimensionality of the observable vector $o$. HMMs of phonemes have three meaningful states and two “hollow” states intended for concatenation of the models of phonemes in the models of words. Each word of the vocabulary is obtained by concatenation of context-independent phonemes (triphones). As the base technology for realization of speech recognizer Hidden Markov Models Toolkit (HTK) [11], developed by the Cambridge University Engineering Department, was used. HTK 3.4 toolkit is employed on all the levels of audio signal processing. Modeling of speech by HTK includes two main stages:

* Training HMMs of acoustical items using a phonetically labeled speech corpus.

* Speech recognition in on-line or off-line modes.

was scored on the isolated signs. Two experiments both composed of two tests were designed. The second test followed the first one after three weeks with identical procedure. Five deaf pupils were chosen from the preliminary class and the first class (5-6 years) as participants for the first experiment and six deaf pupils from the sixth and seventh class (11-13 years) for the second experiment.

<figure><img src="image_32.png" /><figcaption>Fig. 17. An example of two choices from the sheet used for filling answers in multiple-choice test. Three choices are offered in form of three illustrations.</figcaption></figure>

Test material contains the synthesized animation of isolated sign recorded to the video files. 15 signs from videotapes used in the curriculum of the preliminary class were collected, thus the signs would be known by all participated pupils. The sign editor has been used for HamNoSys notation of the signs. Next, we have prepared sheets for the multiple-choice test with three response options (one correct response) composed from randomly arranged pictures of the tested words, Figure 17.

**Procedure** The tested signs were presented on the wall in the classroom. At the beginning of the experiments, five extra non-scored words were presented to demonstrate various options of the study. The size of the signing avatar was approximately 30 cm on the wall. The pupils were not familiar with the tested words before the experiments and the scribing was prevented.

**Results** Tests have been scored as follows. A pupil got one point for the correct answer and no points for the wrong answer. The similarity of some signs was not taken into account. We tested the hypothesis that pupils filled the multiple-choice test by a chance. For this purpose, we have used one-sample and one-sided t-test. The planned comparisons are carried out for both tests of the first experiment and for the first test of the second experiment, ($\alpha$ = 0.01). The results show significantly better understanding of the signed speech than a chance (three options, the chance level 33.3%, p < 0.01). The average 80% success was achieved in the first test of first experiment (t (4) = 6.6, p = 0.0014) and 85% for the second test (t (3) = 17.91, p < 0.001). Better results were achieved in the second experiment with older pupils. There were for the first test on average 95% correct answers (t (5) = 27.59, p < 0.001) and the retesting without the possibility of choosing, on average 80% correct answers. The means of achieved scores are summarized in Table III.
## B. System's training and evaluation

In order to train the speech recognizer a speech corpus was recorded in office conditions using the distant talking directed microphone. About 1000 utterances of two users were recorded and used for training HMMs of phonemes. Totally we have recorded above 10 minutes of speech data from each speaker, these data were labeled semi-automatically in the terms of phoneme sets. It is required to notice that the phonemic alphabets, for instance SAMPA alphabets [12] are rather different for English and Czech, the latter contains more phonemes. Totally 41 different phonemes are used in transcriptions of the lexicon, some words of which have several variants of pronunciations taking into account peculiarities of the speakers. 20 % training utterances were manually labeled by the WaveSurfer software, the rest of the data were automatically segmented by the Viterbi forced alignment method with the flat start [11]. In general the stage of acoustical models training includes the following steps:

* Manual transcription of a lexicon of an applied domain

* Creation of a grammar or a statistical language model (for instance, bi-gram or tri-gram model)

* Preparation of a training speech corpus

* Coding the speech data (feature extraction)

* Definition of topology of HMMs (prototypes)

* Creation of initial HMMs by flat start

* Re-estimation of HMMs parameters of monophones using a labeled speech corpus and Baum-Welch algorithm

* Re-estimation of HMMs of triphones by Baum-Welch algorithm

* Mixture splitting

The speech decoder uses Viterbi-based token passing algorithm [11]. The input phrase syntax is described in a simple grammar that allows to recognize one command in a hypothesis. The audio speech recognizer operates quite fast (less than 0.5xRT) so the result of speech recognition is available almost immediately after detection of speech end by an energy-based voice activity detector. The performance of ASR was evaluated by another speech data, collected in the same office conditions as the training part. The word recognition rate (WRR) was more than 90% on testing phrases pronounced by two male speakers. This rate is acceptable for our task since ordinary single microphone is used for distant speech capturing providing quite low SNR. It was found that SNR for audio signal is under 20 db because of far position (about 2 meters) of the speaker in front of the kiosk and the microphone. In further research a microphone array and corresponding digital signal processing methods for speaker localization and noise elimination are supposed to be applied [13].

## VIII. EVALUATION AND CONCLUSION

Because the kiosk should be usable by all hearing-impaired people (even who cannot read) we designed the GUI in a way that all important text labels are accompanied by an animation of signing avatar with corresponding sign. This new graphical user interface component can be used as a clickable button. This component and whole GUI layout is designed

as a XHTML web page, so that the whole application could be used online, but without ASR and SLR components, and controlled only by a mouse.

The biggest usability problem of the kiosk design is the SL recognition where the user have to start and finish the performed sign in the initial position. This expectation must be explained to the user in the early beginning of the dialogue.

The other parts of the kiosk proved that they can be used without any major usability difficulties. This is achieved by using similar concept of the dialogue as is used in web page browsing which is well-known and the users don't have to think how to control the kiosk.

All modules and scenario configuration are designed in a way that the kiosk can be easily modified to provide another information service or even another type of application (e.g. sign language tutoring tools [1], sign language games or sign language dictionaries [14]).

## ACKNOWLEDGMENT

This project is supported by The Scientific and Technological Research Council of Turkey (TUBITAK) project 107E021, Bogazici University project BAP-03S106.

This research was supported by the Grant Agency of Academy of Sciences of the Czech Republic, project No. 1ET101470416 and by the Ministry of Education of the Czech Republic, project No. ME08106.

We would like to express our great thanks to Zdeněk Krňoul (University of West Bohemia, Pilsen) for contribution to this project by providing 3D avatar for sign language synthesis.
REFERENCES

[1] O. Aran, I. Ari, A. Benoit, A. H. Carrillo, F.-X. Fanard, P. Campr, L. Akarun, A. Caplier, M. Rombaut, and B. Sankur, "Sign Language Tutoring Tool," *Proceedings of eNTERFACE 2006, Summer Workshop on Multimodal Interfaces, Dubrovnik, Croatia*, 2006.

[2] M. Železný, P. Campr, Z. Krňoul, and M. Hrúz, "Design of a multi-modal information kiosk for aurally handicapped people," in *SPECOM 2007 proceedings*, Moscow, Russia, 2005. pp. 751–755.

[3] V. S. V. Vezhnevets and A. Andreeva, "A survey on pixel-based skin color detection techniques," in *Graphicon*, 2003, pp. 85–92.

[4] E. Maggio and A. Cavallaro., "Hybrid particle filter and mean shift tracker with adaptive transition model," in *Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)*, 2005.

[5] O. Aran and L. Akarun, "A particle filter based algorithm for robust tracking of hands and face under occlusion," in *IEEE 16th Signal Processing and Communications Applications (SIU 2008)*, 2008.

[6] R. C. Gonzales and R. E. Woods, *Digital Image Processing*. Prentice-Hall, 2001.

[7] M. Hu, "Visual pattern recognition by moment invariants," *IRE Trans. Inf. Theory*, vol. 8, p. 179–187, 1962.

[8] D. S. Zhang and G. Lu, "A comparative study of three region shape descriptors," in *Proc. of the Sixth Digital Image Computing Techniques and Applications (DICTA02)*, 2002, pp. 86–91.

[9] Z. Krňoul and M. Železný, "A development of Czech talking head," in *Proceedings of ICSPL 2008*, in press, 2008.

[10] R. Rabiner and B. Juang, "Fundamentals of speech recognition," in *New Jersey: Prentice-Hall, Englewood Cliffs, USA*, 1993.

[11] S. Y. et al., "The htk book," in *HTK Version 3.4, Cambridge University Engineering Department*, 2006.

[12] "Sampa alphabet, http://www.phon.ucl.ac.uk/home/sampa/."

[13] S. Brandstein and D. Ward, "Microphone arrays," in *Springer Verlag*, 2000.

[14] J. Langer, "Přínos elektronických výukových pomůcek a slovníků znakového jazyka," In *Vzdělávání sluchově postižných. Praha: MŠMT*, 2006., 2006.

<figure><img src="image_33.png" /></figure>

**Marek Hrúz** was born in 1983 in Slovakia. He received his M.S. degree in cybernetics from the University of West Bohemia (UWB) in 2006. As a Ph.D. candidate at the Department of Cybernetics, UWB, his research interests focus on hand gesture and sign language recognition, particularly tracking and image parametrization, computer vision, machine learning and multimodal human-computer interaction. He is participating in the research project MUSSLAP and is a teaching assistant at the Department of Cybernetics, UWB.

<figure><img src="image_34.png" /></figure>

**Alexey Karpov** Alexey A. Karpov received the M.S. Diploma from St. Petersburg State University of Airspace Instrumentation and Ph.D. degree in computer science from St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), in 2002 and 2007, respectively. His main research interests include automatic speech and speaker recognition, text-to-speech, multimodal interfaces based on speech and gestures, audio-visual speech processing. Currently he is a senior researcher of Speech and Multimodal Interfaces Laboratory of SPIIRAS. He has been the (co)author of more than 60 papers in refereed journals and International conferences, for instance, Interspeech, Eusipco, TSD, etc. His main research results are published by the Journal of Multimodal User Interfaces and by the Pattern Recognition and Image Analysis (Springer). He is a coauthor of the book "Speech and Multimodal Interfaces" (2006, in Russian), and a chapter in the book "Multimodal User Interfaces: From Signals to Interaction" (2008, Springer). He has also been involved in EU SIMILAR Network of Excellence as well as several research projects funded by EU INTAS association and Russian scientific foundations. He was a winner of the 2-nd Low Cost Multimodal Interfaces Software (Loco Mummy) Contest (2006, Brussels). Dr. Karpov is a member of organizing committee of series of International conferences "Speech and Computer" SPECOM, as well as member of the EURASIP, ISCA and OpenInterface associations.

<figure><img src="image_35.png" /></figure>

**Oya Aran** Oya Aran received the BS and MS degrees in Computer Engineering from Boğaziçi University, Istanbul, Turkey in 2000 and 2002, respectively. She is currently a PhD candidate at Boğaziçi University working on dynamic hand gesture and sign language recognition under the supervision of Prof. Lale Akarun. Her research interests include computer vision, pattern recognition and machine learning. She is a student member of the IEEE.

<figure><img src="image_36.png" /></figure>

**Pınar Santemiz** Pınar Santemiz received the B.S. degree in mathematics from Boğaziçi University, Istanbul, Turkey, in 2006. She is currently an M.Sc. student in the Department of Computer Engineering, Boğaziçi University. Her research interests are in the areas of computer vision, sign language analysis, machine learning, and pattern recognition.

<figure><img src="image_37.png" /></figure>

**Pavel Campr** Pavel Campr was born in 1981 in the Czech Republic. He graduated in cybernetics from the University of West Bohemia (UWB) in 2005. As a Ph.D. candidate at the Department of Cybernetics, UWB, his research interests focus on hand gesture and sign language recognition, computer vision, machine learning and multimodal human-computer interaction. He is participating in the research project MUSSLAP. He is also teaching assistant and maintainer of the departmental website.

<figure><img src="image_38.png" /></figure>

**Miloš Železný** was born in Plzeň, Czech Republic, in 1971. He received his Ing. (=M.S.) and Ph.D. degrees in Cybernetics from the University of West Bohemia, Plzeň, Czech Republic (UWB) in 1994 and in 2002 respectively.

He is currently a lecturer at the UWB. He has been delivering lectures on Digital Image Processing, Structural Pattern Recognition and Remote Sensing since 1996 at UWB. He is working in projects on multi-modal speech interfaces (audio-visual speech, gestures, emotions, sign language). He is a member

of ISCA, AVISA, and CPRS societies. He is a reviewer of the INTERSPEECH conference series.
# Design and Evaluation of an Audio-Haptic Interface

Emma Murphy (1), Camille Moussette (2), Charles Verron (3), Catherine Guastavino (1)

(1) McGill University, Canada (2) Umeå Institute of Design, Sweden (3) Orange Labs, France

## Abstract

A non-visual 3D virtual environment, composed of a number of parallel planes has been developed to explore how auditory cues can be enhanced using haptic feedback for navigation. 23 users were asked to locate a target in the virtual structure across both horizontal and vertical orientations of the planes, with and without haptic feedback. In this report, the design of auditory and haptic cues are described and a perceptual evaluation is presented in terms of experimental design, results and analysis.

*Index Terms*—multimodal, haptic feedback, audio feedback, perceptual evaluation.

## I. INTRODUCTION

The main aim of this project was to create a non-visual virtual environment in order to investigate issues of integration and effectiveness of information delivery between audio and haptic modalities. It is proposed that an interface involving a target finding task with audio and haptic (touch) feedback is relevant to this aim. Various studies have indicated that the use of non-speech audio and haptics can help improve access to graphical user interfaces [8, 10], by reducing the burden on other senses, such as vision and speech. Furthermore a mixture of sound and haptic feedback has been found to assist visually impaired users by aiding navigation through virtual environments [11, 16].

There is a substantial body of literature from the field of interface design for musical expression that can inform multimodal interactions for user interface design. Although this discipline is concerned with designing interactions for creative purposes, the issues of mapping between audio and haptic modalities are relevant to user interface design. For example, both gesture-based control of sound synthesis [14] and natural interactions between audio and haptic modalities [1] are also central issues for interaction design in HCI. Wanderly and Orio [13] have made direct parallels between musical tasks and tasks in HCI: "In particular, target acquisition may be similar to the performance of single tones (acquiring a given pitch as well as a given loudness or timbre), while constrained motion may be similar to the performance of specific phrase contours" [13].

Furthermore Marentakis and Brewster [5] have investigated the process of using gesture to control 3D sound for purposes of information display. In this experimental study, time and accuracy ratings indicate that deictic gesture interaction with a

spatial audio display is a robust and efficient interaction technique. Studies have also investigated the combination of audio and haptics to convey object location in non-visual spatial structures [15, 4, 6]. In terms of target acquisition tasks previous research has investigated the addition of haptics to visual interfaces for target-finding [9,12]. More recently Kim and Kwon [3] implemented a haptic and audio grid in order to enhance recognition for ambiguous visual depth cues. The authors implemented a haptic vertical grid and pitch variation to convey a target location to users and subsequent evaluations revealed that the multimodal cues increased precision, particularly in the z-axis [3].

The design of both audio and haptic cues for the virtual interface presented in this study have been informed by previous literature and research. In the following sections, the design of auditory and haptic cues are described and the user evaluation is presented in terms of experimental design, results and analysis.

## II. AUDIO-HAPTIC INTERFACE

<figure><img src="image_39.png" /><figcaption>Figure 1: Virtual environment composed of a number of parallel planes positioned either horizontal or vertical to the phantom stylus.</figcaption></figure>

A virtual environment was created composed of a number of parallel planes of the same dimensions that could be navigated by moving between planes or remaining on the one plane. The virtual structure could be rotated in space so that the planes were either horizontal or vertical (see figure 1). A virtual target was located at a random location on one of the planes. In order to find the target the user first has to navigate through the planes and identify which plane contains the target and then use the auditory cues to locate it. The design idea behind this virtual structure was to create a structure that could be navigated in 3D space. Practical applications of the plane structure include the design of a menu structure or browser for a restricted visual space or non-visual interface.
## A. Audio

Auditory cues were integrated with the haptic environment using MAX/MSP. While the design and evaluation for this interface was not based on a musical instrument, the sounds were informed by a musical design. The idea was realised as a way to complement the haptic movement through the planes as a plucked sound and on a plane as a smooth bowing sound.

<figure><img src="image_40.png" /><figcaption>Figure 2: 3D Sound Mappings</figcaption></figure>

The sound samples were cello recordings (retrieved from www.freesound.org/). Each plane was assigned a plucked pitch corresponding to the first 5 notes of a major scale mapped to the ascending plane number (A3, B3, C#3, D3, E3). The plucked sound was intended to complement the haptic gesture as the users pushed through magnetic feedback on the planes. The bowed sound location cue was intended to complement to the smooth haptic texture while the user navigated the planes. The pitch of the bowed location cue was the same as the plucked sound for that plane level. When the user successfully located a target a distinctive bowed chord auditory cue was played.

## B. 3D Audio Rendering

The bowed sound location cue was spatialised using binaural synthesis with the “ears in hand” technique similar to that reported in [7]. Let (O,X,Y,Z) be the fixed reference coordinate system for both configurations (horizontal and vertical planes). The position of the target is calculated in a system (x,y,z) which translates in (O,X,Y,Z) with its origin attached to the stylus (see figure 2). Let $\theta\phi$ be the spherical coordinates of the target in (x,y,z). The virtual sound source (the bowed cello sound) is spatialised in the same direction in a coordinate system attached to the head of the listener.

The spat object from IRCAM for MAX/MSP was used for binaural sound rendering over headphones [2] Directional cues ($\theta\phi$) are simulated with Head Related Transfer Function (HRTF) filtering based on KEMAR measurements. Reverberation is also used in the object to enhance the externalisation of the virtual source (Jot, 1999). The distance cue is simulated by a 6 dB attenuation of the direct sound when the target-stylus distance doubles. Let $d0$ be a reference distance of 4 cm, for which the virtual sound source level is calibrated to approximately 50 dB-SPL, the gain $g$ of the direct sound is given by:

$$ \text{if } d>d0 \text{ then } g = d0/d \text{ else } g=1 $$

The virtual sound source is played only when the target and the stylus are located on the same plane (horizontal or vertical, according to the configuration). Consequently, in the

horizontal configuration, the elevation of the virtual sound source is 0 degree and the azimuth varies continuously between -180 to +180 degrees; in the vertical configuration, the azimuth of the virtual source is either -90 or +90 degrees, and the elevation varies continuously between -90 and +90 degrees.

## C. Haptics

The haptic effects were designed using H3D, an open source haptic graphic API based on X3D. The experiment was conducted using a Phantom Omni haptic device from Sensable Technologies Inc. This unit offers 3 degrees of freedom with a stylus-type grip and provides a workspace area of 160 W x 120 H x 70 D mm. The Phantom Omni can produce a maximum exertable force of 3.3 Newtons.

The virtual scene created for this interface contained five support planes equally spaced across the device's working area plus a target, a 3D spherical object positioned on one of the plane. A magnetic effect was used to create rigid surfaces to hold users to the planes while they were navigating. The magnetic effect consisted of forces generated in order to keep the proxy (device) on the surface. If the proxy is pulled outside a specific delta distance from the surface it will be freed from the magnetic attraction. The target also had a magnetic force applied, but it should be noted that the forces were weak enough to not attract or guide the device towards the target.

The forces were limited to the planes and the target. The scene and planes were large enough to totally filled the workspace area. Naturally the users were limited also to the mechanical limits of the device in all three axes.

# III. PERCEPTUAL EVALUATION

## A. Experimental Design

The aim of the evaluation of the experiment was to compare the differences for usability and navigation using the two different orientations of the virtual structure; horizontal and vertical, and the two different types of feedback: audio and audio-haptic. Specifically, we investigated whether it would be possible to navigate the interface without the support of the haptic planes. Due to the fact that the interface was not designed with redundant information between modalities we hypothesised that it would be more difficult for users to navigate the environment without haptic feedback.

The experimental design was based on the following independent variables;

* Feedback: audio only, audio-haptic

* Orientation of the virtual planes: vertical, horizontal

This resulted in four experimental conditions:

1. Audio-Haptic Vertical

2. Audio-Haptic Horizontal

3. Audio-Only Vertical

4. Audio-Only Horizontal
The following dependent variables were integrated into the
experimental design:

* Completion times; time taken for the user to navigate through the planes and find the target

* Trajectories; path travelled by user from the starting point of the stylus to the target

* Perceived effectiveness and ease of use of both audio and haptic cues

* Cognitive strategies reported by participants to navigate the virtual structure and located the target.

## B. *Procedure*

23 participants volunteered for the study, 20 males and 3 females between the ages of 24 and 55. Participants were either involved in the eNTERFACE '08 project or part of the wider research community at LIMSI.

Participants were provided with both a short training introduction and a practical trial session before beginning the main experiment. The initial training introduction involved presenting a visual description of the virtual structure. While the entire experiment was non-visual it was considered useful to show the users a visual representation of the virtual structure so that they could visualise the task mentally. Furthermore for many of the participants, English was not a first language and therefore the provision of a visual representation ensured that users were not only relying on the experiment protocol instructions delivered through English. During this introductory session users became familiar with some of the audio cues using a visual representation of the planes. It was considered relevant to train the users on the sounds to convey changing planes, locating the target and the confirmation sound when the target was successfully hit. However, users were not given any information about the 3D audio mappings or the haptic feedback.

In a practical training session, users were presented with 8 trials, 2 of each condition. Users were asked to navigate the planes, firstly find the plane with the target and then locate that target. For the main experiment participants were presented with 44 Trials (11 per condition)

The condition and target position within and across planes were randomised. In order to move onto the next trial participants were asked to press a button on the phantom stylus when they were ready to move on. Timings were recorded from this point until the user located the target. User trajectories across and within the virtual planes were also recorded. Participants were also asked to complete a post-task questionnaire concerning perceived effectiveness and ease of use of the audio-haptic cues and the participant's cognitive strategies for finding the target.

## C. *Participants*

23 participants volunteered for the study, 20 males and 3 females between the ages of 24 and 55 (mean age: 32, SD: 8.2). Participants were either involved in the eNTERFACE

'08 workshop or part of the wider research community at LIMSI, CNRS, France, involved in audio, computer science or engineering related research.

## D. *Procedure*

Participants were provided with both a short training introduction and a practical trial session before beginning the main experiment. The initial training introduction involved presenting a visual description of the virtual structure. While the entire experiment was non-visual it was considered useful to show the users a visual representation of the virtual structure so that they could visualise the task mentally. Furthermore for many of the participants, English was not a first language and therefore the provision of a visual representation ensured that users were not only relying on the experiment protocol instructions delivered through English. During this introductory session users became familiar with some of the audio cues using a visual representation of the planes. It was considered relevant to train the users on the sounds to convey changing planes, locating the target and the confirmation sound when the target was successfully hit. However, users were not given any information about the 3D audio mappings or the haptic feedback.

In a practical training session, users were presented with 8 trials, 2 of each condition. Users were asked to navigate the planes, firstly find the plane with the target and then locate that target. For the main experiment participants were presented with 44 trials (11 per condition), except for the first 3 participants who completed 40 trials (10 per condition). The condition and target position within and across planes were randomised. In order to move onto the next trial participants were asked to press a button on the phantom stylus when they were ready to move on. Timings were recorded from this point until the user located the target. User trajectories across and within the virtual planes were also recorded. Participants were also asked to complete a post-task questionnaire concerning perceived effectiveness and ease of use of the audio-haptic cues and the participant's cognitive strategies for finding the target.

# IV. INITIAL RESULTS

## A. *Completion Times*

Users successfully located the target across all trials for the audio-haptic condition. In the audio-only condition, 4 users failed to locate the target for a combined total of 10 trials (6 horizontal, 4 vertical). Their timings were recorded for the unsuccessful trials (AV 167s) but not included in the data for completion times.
<figure><img src="image_41.png" /><figcaption>Figure 3: Mean completion times for vertical and horizontal orientations across both audio only and audio-haptic conditions</figcaption></figure>

Furthermore, 62 trials out of 1000 trials were excluded from the data analysis due to technical difficulties. A repeated measures factorial ANOVA for completion times revealed significant effects of orientation and feedback. The time taken for users to complete the task in the horizontal condition was significantly less than that of vertical condition (F(1, 936) = 6.1, p=0.02) (see figure 3).

In addition, as illustrated in figure 3, completion times for the audio-only condition were significantly longer than that of the audio-haptic condition (F(1, 936) = 49.3, p<0.001). Furthermore, post-hoc tests revealed a significant difference between the vertical and horizontal orientations for both audio-only (t(243) = -2.94, p two-tailed =0.004) and audio-haptic conditions (t(228) = -3.07, p two-tailed =0.002). There were no interaction effects between feedback and orientation.

## B. *Qualitative Post task Feedback*

In the post-task questionnaires users were asked to describe their strategies for finding the target using the multimodal cues. Most users described a process of first navigating the virtual space to determine the orientation of the planes, then using the auditory cues to first determine the correct plane and concentrate on the location cue to find the target. A further analysis of qualitative user comments will be conducted in conjunction with an analysis of recorded user trajectories.

## V. CONCLUSION AND FUTURE WORK

The focus of this project was to enhance an auditory navigation task in a non-visual virtual environment though haptic cues. Evaluation experiments revealed that the audio-haptic feedback condition was significantly less difficult for users to navigate. Furthermore, the horizontal orientation was easier to navigate for both audio-haptic modalities. Following a more detailed analysis of results from the experiment presented in this report, we intend to develop this study in terms of the audio-haptic cues, and also using other haptic devices.

## VI. ACKNOWLEDGEMENTS

This study was supported by an NSERC-SRO team grant on ENACTIVE Interfaces (P.I.: M. Wanderley) and took place during the eNTERFACE’08 workshop at LIMSI, CNRS, France. We would like to thank Orange Labs for providing the haptic device, all of the Interface and LIMSI staff and participants that volunteered for the study, Dr. Antoine Gonot for helpful discussion on navigation with auditory cues, and Dr. Ilja Frissen for his help and advice in the analysis of quantitative results.

## REFERENCES

[1] Essl, G. & O'Modhrain, S. 2006. An enactive approach to the design of new tangible musical instruments. Organised Sound 11(3), pp. 285-296.

[2] Jot, J. 1999, "Real-time Spatial processing of sounds for music, multimedia and interactive human-computer interfaces," ACM Multimedia Systems J. 7(1).

[3] Kim, S., Kwon, D. Haptic and Sound Grid for Enhanced Positioning in a 3-D Virtual Environment, Haptic and Audio Interaction Design, Second International Workshop, HAID 2007, Seoul, South Korea, November, pp. 29-30.

[4] Lahav, O. & Mioduser, D., 2004. Blind persons' acquisition of spatial cognitive mapping. Sharkey P., McCrindle R. & Brown D. (Eds) Proceedings of 2004 International Conference on Disability, Virtual Reality and Associated Technologies; September 20-22; Oxford, UK, pp. 131-138.

[5] Marentakis, G. & Brewster, S.A.(2005). Gesture interaction with spatial audio displays: Effects of target size and inter-target separation. Proceedings of the 2005 International Conference on Auditory Display; July 6-9; Limerick, Ireland, pp. 77-84.

[6] Murphy, E., Kuber, R., Strain, P., McAllister, G. & Yu, W., 2007. Developing Sounds for a Multimodal Interface: Conveying Spatial Information to Visually Impaired Web Users. Proceedings of the 2007 International Conference for Auditory Display; June 26-29; Montreal, Canada.

[7] Magnusson, C., Danielsson, H & Rassmus-Gröhn, K., 2006. Non Visual Haptic Audio Tools for Virtual Environments, Haptic and Audio Interaction Design, First International Workshop, Glasgow, Scotland, 31st August - 1st September.

[8] Mynatt, E. & Weber, G., 1994. Nonvisual presentation of graphical user interfaces: contrasting two approaches. Adelson, B., Dumais, S. T. & Olson J. S. (Eds) Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; April 24-28; Boston, Massachusetts, United States. New York: ACM Press, pp. 166-172.

[9] Oakley, I., McGee, M. R., Brewster, S. & Gray, P., 2000. Putting the feel in look and feel. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '00), pp. 415-422.

[10] Ramstein, C., Martial, O., Dufresne, A., Carignan, M., Chasse, P. & Mabilleau, P., 1996. Touching and hearing GUIs: Design issues for the PC-access system. Proceedings of the 1996 Annual ACM Conference on Assistive Technologies; April 11-12; Vancouver, British Columbia, Canada. New York, ACM Press, pp. 2-9.

[11] Sjöström, C., 2001. Designing haptic computer interfaces for blind people. Proceedings of the 2001 International Symposium on Signal Processing and its Applications; August 13-16; Kuala Lumpur, Malaysia, pp. 68-71.

[12] Wall, S. A., Paynter, K., Shillito, A. M., Wright, M. & Scali, S., 2002. The effect of haptic feedback and stereo graphics in a 3d target acquisition. In Proceedings of Eurohaptics, pp. 23-29.

[13] Wanderley, M. & Orio, N., 2002. Evaluation of input devices for musical expression: Borrowing tools from HCI. Computer Music Journal, 26(3) pp. 62-76.

[14] Wanderley, M. M. & Depalle, P., 2004. Gestural control of sound synthesis. Proceedings of the IEEE, Johannsen, G. (Ed.) Special Issue on Engineering and Music - Supervisory Control and Auditory Communication, 92(4), pp. 632-644.

[15] Wood, J., Magennis, M., Francisca, E., Arias, C., Graupp, H., Gutierrez, T. & Bergamasco, M., 2003. The design and evaluation of a computer
game for the blind in the GRAB haptic audio virtual environment.
Proceeding of EuroHaptics 2003; July 6-9; Dublin, Ireland.

[16] Yu, W. & Brewster, S. (2002). Multimodal virtual reality versus printed medium in visualization for blind people. *Proceedings of the 2002 International ACM conference on Assistive technologies; July 8-10; Edinburgh, Scotland.* 57-6

**Emma Murphy** is currently a post-doctoral research fellow at the Multimodal Interaction Lab, McGill University, Montreal. She is an honours graduate of Trinity College, Dublin, where she studied Music and Philosophy. Since graduating from TCD in 2000, her career path has followed from full-time involvement in research and development in industry to academic study. Emma completed an MSc in Music Technology in 2003 at the Centre for Computational Musicology and Computer Music, University of Limerick. In 2007, she completed and successfully defended her doctoral thesis entitled "Designing Auditory Cues for a Multimodal Web Interface: A Semiotic Approach" at the Sonic Arts Research Centre, Queen's University Belfast.

**Camille Moussette** Born in Montreal, Canada. Hyperactive by nature, Camille Moussette likes the blurry connections between humans, atoms and bits. He holds a Physics degree, a bachelor in Industrial Design and a Masters in Interaction Design. His work experience includes microelectronics R&D (IBM and NRC Canada), web consulting and various involvement in projects ranging from building architectural snow structures in Scandinavia and Italy, to developing new mobile systems in collaboration with Nokia. He is currently pursuing a Ph.D. researching mobile haptic interfaces and teaching classes in Experience Prototyping, Hardware Sketching and Design Ethnography at the Umeå Institute of Design in Sweden.

**Chareles Verron** Charles Verron was born in Caen, France, in 1980. He received an engineering degree from ENSIETA (Brest, France) in 2003, and a master's degree in acoustics, signal processing and computer sciences applied to music from IRCAM (Paris, France) in 2004. In 2006 he was a research assistant in the Computer and Audio Research Laboratory at Sydney University, Australia. He is currently doing a Ph.D. at Orange Labs, France. His research interests include sound synthesis and sound spatialization.

**Catherine Guastavino** Dr. Guastavino is a Professor at McGill University (Canada). She holds a Ph.D. in Psychoacoustics (Université Paris 6, France), and received post-doctoral training in the fields of memory and cognition at McGill, before joining the McGill School of Information Studies in 2005. Dr. Guastavino also serves as Associate Director of the Centre for Interdisciplinary Research on Music Media and Technology (CIRMMT), an inter-institutional research center based in Montreal, and is an associate member of the McGill Schulich School of Music. She investigates what information can be best conveyed using sound or touch as opposed to relying on text and graphics.
# Capture and machine learning of physiological signals

Benedicte Adessi(1), Rami Ajaj(2), Carole Arrat(3), Ivan Chabanaud(4), Matthieu Courgeon(2), Nicolas Déflache (3), Nesli Erdogmus(4), Fabienne Gotusso(5), Hikmet Gökhan Himmetoğlu(6), Christian Jacquemin(2), Loïc Kessous(2), Jean-Claude Martin(2), Michal Osowski(7), Thomas Pachoud(8), Jana Trojanova(9)

(1) Independent Arttist, Marseilles, (2) LIMSI-CNRS, Orsay, France, (3) ESAV, Toulouse, (4) Independent Arttist, Marseilles, (3) Independent Sound Engineer, Paris, (4) METU - Electrical and Electronics Engineering, Ankara, Turkey, (5) Independent Arttist, Paris, (6) Koç University, Istanbul, Turkey, (7) Independent Arttist, Paris, (7) Independent Arttist, Amsterdam, (8) Independent Arttist, Paris, (9) Department of Cybernetics, University of West Bohemi, Czech Republic

## Abstract

The topic of the eNTERFACE project #5 concerned emotion recognition. Its purpose was to design and experiment a set of components that would relate input signals to emotions, and use it as clue to generate correlated sound or graphics.

Index Terms—affective computing, physiological signals, emotions

## I. OVERVIEW

The topic of the eNTERFACE project #5 concerned emotion recognition. Its purpose was to design and experiment a set of components that would relate input signals to emotions, and use it as clue to generate correlated sound or graphics (Figure 1). Two types of input signals have been considered: signals captured by physiological sensors (heart beat and breath) and live images. As for the analysis, several tools have been used: statistical measures, neural networks, image processing, and clustering. The interfaces between the modules have been a focus of attention for a better connection when the developments are completed. Several output modalities have been considered: live music, live graphics, and particles systems.

Because of the diversity of the participants, we found it useful to have regular presentations so that we could discover the member's scientific interests. 11 presentations have been made in the morning and have covered a wide variety of topics: real-time video processing, live 3D graphics, facial feature extraction, emotions and facial animation, sonification from sensor data, 3D audio through WFS, facial animation, audio signal analysis, and tactile interface.

Due to the importance of the task, the whole processing sequence has not been completed, but several modules have however been realized and could be demonstrated autonomously by simulating the input they were supposed to receive. Since participants had so diverse background such as performance or laboratory, some being students, others professionals, artists or scientists, since some were discovering sensors and others were experienced, we found it useful to have regular presentations so that we could discover the member's interests. The participants have however gained a rich experience from the scientific exchanges through the collaborative design of components, through the regular

morning presentations, and through informal talks with participants from the other groups.

The anticipated inputs for the correlated sound and graphics generation were ‘breath signal’, ‘heart beat’, ‘voice’ and ‘live images of face’. All of those modules were realized except for the ‘voice’ due to the lack of a specialized person in the team on this subject. Detailed information on the feature extraction from the physiological signals and face images is given in the next section.

<figure><img src="image_42.png" /><figcaption>Figure 1 Overview of the system for emotion recognition</figcaption></figure>

## II. FEATURE EXTRACTION

Extraction of features from signals has been done to obtain meaningful parameters for the control of the audio-visual generators.

### A. Breath Signal

On the signal of the nasal respirometer, we extracted different statistical features. Then using a real-time algorithm for local maxima extraction we compute both explicit features and statistical features. By hypothesis, as the sensor measure the temperature at the output of the nose, we consider that the expiration phase corresponds to increasing of the sensor value (temperature) and decreasing value to the inspiration phase. Following this principle, we can extract several features that are relative to range and duration of the inspiration, expiration and full respiration cycles. In a second step, we extracted 5 statistical features from each of the previous features. These 5 features are computed on N successive values of each previous
feature. Using a 'window' of N values, we extracted the minimum, the maximum, the mean, the standard deviation, the skewness (measure of the asymmetry of the data around the sample mean) and the kurtosis (measure of how outlier-prone a distribution is) of each feature.

## B. Heart Beat

The ECG signal is the manifestation of contractile activity of the heart. It can be used to measure heart rate (HR), inter-beat intervals (IBI) and to determine the heart rate variability (HRV). In our project, we just used the heart rate that was extracted from the ECG signal as a component for emotion recognition.

## C. Live Images of Face

Live face image analysis has also been considered as a complementary means for emotion recognition. There are two main approaches for feature extraction from images: holistic methods and geometric methods. We are using a geometric method proposed by Cootes [3]. The method is mostly known under acronym ASM (Active Shape Model). Its basic idea is as follows: ASM is a statistical model which is able to deform to fit to a similar shape as the ones who were presented during training period (see Figure 2).

The main problem in ASM is the localization of the points of the model. There were several enhancement since the ASM was introduced. We would like to implement the LBP (Local binary patterns) to improve the fitting of the model.

### Model is represented:

<figure><img src="image_43.png" /></figure>

### Training
(we obtain mean shape and transformation func.)

<figure><img src="image_44.png" /></figure>

### Fitting
 (actual shape and deformation from the mean shape)
<figure><img src="image_45.png" /></figure>

In order to map our N-dimensional feature vector to the two dimensional emotion space, Stuttgart Neural Network Simulator modified for Java (JavaNNS) is used. JavaNNS is a simulator for artificial neural networks, i.e. computational models inspired by biological neural networks. It enables the user to use predefined networks or create your own, to train and to analyze them.

The designed network that takes physiological features as input and gives the corresponding emotion point (A,V), is first trained with a labeled training set. After the training is completed, the model is exported out as a C++ function that can easily be mounted into the system.

Unfortunately, there are many difficulties in the process. One of the challenging points here is to extract right features that contain the distinguishing data on emotions and to label these feature vectors correctly. In addition to that, the right structure for the network has to be built. Back-propagation method for training is proven to be fast and accurate enough in most cases and thus it can be used in real time processing. For activation and output functions of the input and output nodes and the hidden nodes, many comprehensive tests and trials have to be conducted with the real data.

## V. INTERPRETATION OF SENSOR DATA

Emotion is a complex phenomenon that no conceptual theory describes at a sufficient level at the moment. It involves percepts as well as cognition (concepts), conation (impulse) & affects (feelings, physiological systems). Everyday language provides a manageable simplification of concepts & experiences.

Emotional life is a large and complex space that we can't represent completely, because of technical, conceptual and experimental limitations.

* Technical: we need to be aware of what the multimodal inputs we have are capable of describing. We realize we have few parameters and they are superficial.

* Conceptual: are we interested in basic emotions that statistically occur more often (scientific approach), or are we going to fit subjective needs (artistic approach) ?

* Experimental: there is no obvious link, and we don't have knowledge to link measurements to the inner emotion of the human being sensed.

A subject found in a measured state equivalent to a state in which he was before does not mean that he is going through

the same particular emotion. Inversely, being in a particular emotional state could result in a comparable measure, even though it is not necessarily the case. Anyway, it would require one step more of analysis to link the representation to an inner emotion.

So, what is represented and what is not? Are we recognizing emotions, are we recognizing concepts, or are we recognizing face expressions + sensor features?

With an artistic approach, it is possible to rely on the feedback of the multimedia result on the performer's perception to create a link between capitation and actual inner emotion. If expressivity is placed at a sufficient level to allow this loop to exist, the performer addresses directly to the multimedia content that includes things we wouldn't normally accede, because the performer isn't even conscious about. Because it is explicit, this is the only space where we know where we are.

Thus, we need a system of representation with emphasis on experiential description.

Intuitively, we proposed to use the Activity / Valence representation, which we assume to be a broad enough concept to fit the view of several artists, or at least to be of interest for them. This representation is not describing the context (environment, cognition, affect...) but taking it into account. Emotional life is divided into 4 poles, 4 poetic worlds that the artists describe with general sentences. An example of a poetic world mapped to sound generators is given in the following figure 4.

<figure><img src="image_46.png" /><figcaption>Figure 4 Poles of emotional life</figcaption></figure>

## VI. ARTISTIC WORK

We now present a few artistic applications that we made during the course of the workshop.
## A. Sound Generation

Breathing and Thoracic Volume: For breathing we used the sensor signal to modulate a filtered noise. We used a white noise and a band-pass filter with a moderate coefficient of resonance. The signal from the breath temperature sensor was connected to both the amplitude of the output signal and the center frequency of the filter. We apply a flanging effect at the output of the filter in order to give coloration to the resulting sound. For the thoracic volume we used a similar method but with a voiced source as signal input to feed the filter.

A toolbox for Heart beat sonification: We designed a toolbox for Heartbeat sonification using two different methods. These methods have different levels of flexibility, in particular, concerning spectral transformation. The first system uses complex resonances. Using parallel resonance filter bank based on the CNMAT tools, we made a module which allows us to synthesize an arbitrary resonance model and to control and modify several high-level parameters. The sonic interaction designer can control and modify the number of resonances and the lower resonance frequency. This model can be modified by changing other parameters at a high-level control as the global resonance and the spectral amplitude slope. A second module was build to use sampling techniques. This module use a polyphonic player that allows playing recorded sounds by triggering it and avoids clicks due to a re-triggering when it's already playing a sound, by allocating a new playing voice to it. Modifiable parameters are speed (pitch modified), time of beginning for reading into the sound file and parameters of a linear amplitude envelope defined by fade in time and fade out time.

## B. Live performances based on input signals

The point is to create a sound-space worked by the variation of the emotions felt during the act of performing. Michal Osowski wishes to build an « emotional curve » from analyzing the video of the performance and then to program automatic sounds on the computer that happen in relation to the variations worked by the curve.

The performance is shot by the camera of Ivan Chabanaud. Fabienne Gotusso creates the performance choosing as a point of departure to the improvisation to play with the sound of the German word « Holzwege » (meaning forest path) and the English word « to die ». The aim is to create some estrangement from the meaning of both words and then to liberate the games and associations of meaning produced by the pronunciation of the words relating to improvised acts. She chooses to improvise in a sandy path with a tree standing by. Then Michal and Fabienne manage to extract some range of emotions by watching the video of the performance to finally order them as a map.

Fabienne dresses in a black suit on which several captors are fixed. She begins to move after several necessary preliminary measures in order to identify her skeleton. Skeleton that will appear at the time on the computer screen as she is acting the performance. The skeleton reproduces in real time the motions composed by Fabienne. The inscription and architecture of the

motions are plainly faithful to the quality involved in the dancing. It is impossible not to think about E J Marey's works.

## VII. CONCLUSION

Even though this workshop has not allowed the participants to design a full working system from signal capture to emotion recognition and reactive digital art, it has however had numerous positive achievements:

* through regular seminars, participants have discovered new methods and new techniques in signal analysis, machine learning, computer graphics and sound, live performance, and affective computing,

* through joint developments, the researchers have gained a better insight on the requirements of physiological signal analysis and interpretation,

* through human experience and joint collaboration, the participants have discovered shared scientific interests and defined new perspectives for future collaborative works.

## Acknowledgement:

Part of this research was supported by the Grant Agency of Academy of Sciences of the Czech Republic, project No. 1ET101470416.

## REFERENCES

[1] T. Jehan, A. Freed, and R. Dudas. "Musical applications of new iter extensions to Max/MSP", Proceedings of the International Conference on Computer Music, pages 504-507, 1999.

[2] R Cowie, "Emotional life: Terminological and conceptual clarifications", HUMAINE - D3i, 2006 http://emotion-research.net/projects/humaine/deliverables/D3i%20final.pdf

[3] Cootes, T. F., Edwards, G. J., and Taylor, C. J. 2001. Active Appearance Models. IEEE Trans. Pattern Anal. Mach. Intell. 23, 6 (Jun. 2001), 681-685. DOI= http://dx.doi.org/10.1109/34.927467
# Multimodal Feedback from Robots and Agents in a Storytelling Experiment

S. Al Moubayed (#), M. Baklouti (^), M. Chetouani (*), T. Dutoit (+), A. Mahdhaoui (*), J.-C. Martin (~), S. Ondas (@), C. Pelachaud (°), J. Urbain (+), M. Yilmas (&)

(#)Center for Speech Technology, Royal Institute of Technology, KTH, SWEDEN, (^) Thalès, FRANCE, (*) University of Paris VI – FRANCE, (+) Faculté Polytechnique de Mons – BELGIUM, (~) LIMSI – FRANCE, (@) Technical University of Košice – SLOVAKIA, (°) INRIA – FRANCE, (&) Koc University – TURKEY

**Abstract** — In this project, which lies at the intersection between Human-Robot Interaction (HRI) and Human-Computer Interaction (HCI), we have examined the design of an open-source, real-time software platform for controlling the feedback provided by an AIBO robot and/or by the GRETA Embodied Conversational Agent, when listening to a story told by a human narrator. Based on ground truth data obtained from the recording and annotation of an audio-visual storytelling database, and containing various examples of human-human storytelling, we have implemented a proof-of-concept ECA/Robot listening system. As a narrator input, our system uses face and head movement analysis, as well as speech analysis and speech recognition; it then triggers listening behaviors from the listener, using probabilistic rules based on the co-occurrence of the same input and output behaviors in the database. We have finally assessed our system in terms of the homogeneity of the database annotation, as well as regarding the perceived quality of the feedback provided by the ECA/robot.

## I. INTRODUCTION

This project lies at the intersection between Human-Computer Interaction (HCI) and Human-Robot Interaction (HRI).

Human-Robot Interaction (HRI) is a multi-disciplinary field involving research on robot control (planning, sensor...), speech processing (recognition, synthesis), vision (human localization, environment characterization), artificial intelligence, cognitive science and other fields [1]. Various robots are now available for such studies, and are provided with specific programming tools. In this project, we have focused of the Sony AIBO dog, and the URBI (Universal Real-time Behaviours Interface) language [2].

Human-Computer Interaction is restricted here to Embodied Conversational Agents (ECAs). The term ECA

has been coined in Cassell et al. [3] and refers to human-like virtual characters that typically engage in face-to-face communication with the human user. In this project, we have used GRETA [4], an ECA, whose interface obeys the SAIBA (Situation, Agent, Intention, Behavior, Animation) architecture [5].

Several methods have been proposed for the improvement of the interaction between humans and agents or robots. The key idea of their design is to develop agents/robots with various capabilities: establish/maintain interaction, show /perceive emotions, dialog, display communicative gesture and gaze, exhibit distinctive personality or learn/develop social capabilities [6, 7, 8]. Theses social agents or robots aim at naturally interacting with humans by the exploitation of these capabilities.

We have investigated one aspect of this social interaction: the engagement in the conversation [9]. The engagement process makes it possible to regulate the interaction between the human and the agent or the robot. This process is obviously multi-modal (verbal and non-verbal) and requires an involvement of both the partners. Some mechanisms as motivation, curiosity can be useful for this purpose [8].

Our project more specifically aims at exploring multimodal interaction between a human speaker telling a story (typically a cartoon) to (i) an ECA or (ii) an AIBO robot. More particularly, we focused on the design of an open-source, real-time software platform for designing the feedbacks provided by the robot and the humanoid during the interaction. The multimodal feedback signals we consider here are limited to facial and neck movements by the agent, while the AIBO robot uses all possible body movements, given its poor facial expressivity. We did not pay attention to arms or body gestures.

This paper is organized as follows. In Section II, we formalize SAIBA, a common architecture for embodied agents, and introduce its application to feedback modeling. This leads us, in Section III, to exposing the contents of the eNTERFACE08_STEAD database developed for this project and containing various annotated examples of

<footnote>This report, as well as the accompanying database and the source code for the software developed during the project, are available online from the eNTERFACE'08 web site: www.interface.net/interface08.</footnote>
human-human storytelling. This database was used for designing several feedback components in subsequent Sections. Sections IV and V respectively focus on the speech and video analysis modules we have used. This is followed in Section VI by details on possible control of the agent state via ASR. We then give, in Section VII, a description of the feedback rules we have established for triggering feedback from our software and hardware rendering engines, namely AIBO and GRETA, and show the behaviors we have been able to synthesize with them. Finally, Section VIII gives details on the compared performances of our HCI and HRI systems.

## II. SAIBA AND FEEDBACK

Although we are all perfectly able to provide natural feedback to a speaker telling us a story, explaining how and when you do it is a complex problem. ECAs are increasingly used in this context, to study and model human-human communication as well as for performing specific automatic communication tasks with humans).

Examples are REA [10], an early system that realizes the full action-reaction cycle of communication by interpreting multimodal user input and generating multimodal agent behaviour, the pedagogical agent Steve [11] which functions as a tutor in training situations, MAX [12] a virtual character geared towards simulating multimodal behaviour, Gandalf [13] provides real-time feedback to a human user based on acoustical and visual analysis. Carmen [14] a system that supports humans in emotionally critical situations such as advising parents of infant cancer patients. Other systems realize presentation agents [15], i.e. one or more virtual agents present some information to the user. They can adopt several roles, such as being a teacher [15; 17], a museum guide [18, 19, 12] or a companion [20, 21]. In robotics, various models have been proposed for the integration of feedbacks during interaction [7]. Recently, the importance of feedbacks for discourse adaptation has been highlighted during an interaction with BIRON [22].

In a conversation, all interactants are active. Listeners provide information to the speaker on how they view the conversation goes on. By sending acoustic or visual feedback signals, listeners show if they are paying attention, understanding or agreeing with what is being said. Taxonomies of feedbacks, based on the meaning these signals convey, have been proposed [23, 24]. The key idea of this project is to automatically detect the communicative signals in order to produce a feedback. Contrary to the approach proposed in [LOH08], we focus on non-linguistic features (prosody, prominence) but also on head features (activity, shake, nod).

Our system is based on the architecture proposed by [4], but progressively adapted to the context of a storytelling (figure 1). We developed several modules for the detection and the fusion of the communicative signals from both audio and video analysis. If these communicative signals match our pre-defined rules, a feedback is triggered by the

Realtime BackChannelling module, resulting in two different behaviors conveying the same intention.

<figure><img src="image_47.png" /><figcaption>Fig. 1. Architecture of our interaction feedback model</figcaption></figure>

## III. THE ENTERFACE08_STEAD DATABASE

<figure><img src="image_48.png" /></figure>

In order to model the interaction between the speaker and the listener during a storytelling experiment, we first recorded and annotated a database of human-human interaction: the eNTERFACE08_STEAD database. This database was used for extracting feedback rules (section VII, but also for testing the multi-modal feature extraction system (section VIII)).

We followed the McNeill lab framework [25]: one participant (the speaker), who has previously observed an animated cartoon (Sylvester and Tweety), tells the story to a listener immediately after viewing it. The narration is accompanied by spontaneous communicative signals (filled pauses, gestures, facial expressions, etc.). In contract, instructions are given to the listener to express his/her engagement in the story by giving non-verbal audio-visual gestures in response to the story told by the speaker.

### eNTERFACE_STEAD Contents

Twenty-two storytelling sessions telling the “Tweety and Sylvester - Canary row” cartoon story were recorded.

Thirteen recording sessions were done by a French listener and a French speaker. The last two recordings have exaggerated non-verbal activity (closer to acting than to real-life storytelling).