Anthropomorphic Agent as an Integrating Platform ofAudio-Visual Information
Shigeki Sagayama

Takuya Nishimoto

Graduate School of Information Science and Technology, The University of Tokyo
Hongo, Bunkyo-ku, Tokyo 113-8656 Japan / {sagayama,nishi}@hil.t.u-tokyo.ac.jp

1

Introduction

Task
Information

In integration of audio-visual information input fromsensors and control output to actuators and roboticsystems, anthropomorphic spoken-dialog agent canbe a good platform combining them under a unified
concept such as “virtual human.” This talk will focus
on a anthropomorphic spoken-dialog agent and its
future possibility as a integrating platform.

2Spoken Dialog Agent

2.1

Galatea Toolkit

One of ultimate human-machine interfaces is anthropomorphic spoken dialog agent which behaves likehumans with facial animation and gesture and makespeech conversations with humans. Among numerous efforts devoted for such a goal, Galatea Project
conducted by 17 members from 12 universities is developing an open-source license-free software toolkit[1] for building an anthropomorphic spoken dialog
agent under a financial support from IPA1 during
fiscal years of 2000–2002. The authors are membersof the project. The features of the toolkit are as follows: (1) high customizability in text-to-speech synthesis, realistic face animation synthesis, and speechrecognition, (2) basic functions to achieve incremental (on-the-fly) speech recognition, (3) mechanism for“lip synchronization”; synchronization between audio speech and lip image motion, (4) “virtual machine” architecture to achieve transparency in module to module communication. The Galatea Toolkit
for UNIX/Linux and Windows operating systems
will be publicly available from August 22, 2003, at
http://hil.t.u-tokyo.ac.jp/∼galatea/.

2.2Toolkit Components

The Galatea Toolkit consists of five functional modules: speech recognizer, speech synthesizer, facial animation synthesizer, agent manager which works as
an inter-module communication manager, and task
(dialog) manager. Fig. 2.2.1 shows the basic module architecture of the Galatea toolkit. The Galatea
Project members newly created these components ormodified existing components of their own or publicly available. The outline of some of these functional modules are stated below.
2.2.1

Common features

TaskInformation

Task Manager

Prototyping Tools

DialogModelDialog
Model

Other
Application
ModuleAgent Manager

SpeechRecognition
Module(SRM)

Face image
Synthesis
Module(FSM)

SpeechSynthesis
Module(SSM)

IIPL

Microphone

CRT

Speaker

Figure 1: System architecture of Galatea.
the model parameters are trained, facial expressions
and voice quality can be controlled easily.
2.2.2

Speech recognition module (SRM)

SRM consists of three submodules: the commandinterpreter, the speech recognition engine, and thegrammar transformer. Based on a speech recognitionengine “Julian” developed by Kyoto University andothers, it accepts the grammar to represent sentencesto recognize and has been modified to accept multiple formats for grammar representation and output
incremental recognition results. It can change thegrammar by request from external modules during
dialog sessions. It also produces N -best recognition
candidates for sophisticated use of multiple results.
2.2.3

Speech synthesis module (SSM)

This module is the first open-source license-free
Japanese Text-to-Speech conversion system consisting of four sub-modules. The command interpreter
receives an input command from the agent manager and invokes sub-processes according to the command. The text analyzer decomposes arbitrary
Japanese input texts containing Kanji, Kana, alphabetic, numeric characters, and optionally embedded
tags according to the JEIDA-62-2000 [3] typicallyspecifying the speaking style, and extracts linguistic information including pronunciation, accent type,part of speech, etc., partly utilizing ChaSen[2] andnewly developed dictionaries for Japanese morphological analysis. The waveform generation engine isan HMM-based speech synthesizer, that simultaneously models spectrum, F0 and duration in a unified
framework of HMM capability. The speech output
sub-module outputs the synthetic speech waveform.

Galatea employs model-based speech and facial animation synthesizers whose model parameters are
adapted easily to those for an existing person if
his/her training data is given. Synthesized facial
images and voices are customizabile easily depending on the purposes and applications of the toolkit
users. This customiazability is achieved by employing model based approaches where basic model parameters are trained or determined with a set of 2.2.4 Facial image synthesis module (FSM)
training data derived from an existing person. Once
FSM is a module for high quality facial image synthe1 Information-Technology Promotion Agency
sis, animation control and precise lip-synchronization

Figure 2: Examples of synthetic expressions from asingle photo.
with synthetic and natural voice. To customize theface model, a graphical user interface is equipped tofit a generic face wire frame model onto a frontal facesnap shot image. Face action units are defined on
this generic model and prototype facial expression
can be synthesized by combination of these action
units. Also autonomous actions such as blinking andnodding can be generated. Lip movement in an utterance is controlled by “viseme” and duration. Facial
animation is expressed easily by a simple script.
2.2.5

Task manager (TM)The task of user-agent dialog management can be described in VoiceXML. TM consists of translator, from
VoiceXML documents to the intermediate language(Primitive Dialogue Operation Commands, PDOC),
and the dialogue controller that interprets the PDOC
documents. We extended the original specification of
VoiceXML to add some commands, including the facial expression controls of anthropomorphic dialogueagents.

3

Integrating Platform

3.1

Galatea Architecture

In the Galatea Toolkit, the functional units are independently modularized, input/output devices are
directly managed in the module, and the agent manager controls inter-module communication. In orderto easily integrate additional modules, all modules
are modeled as virtual machines having a simple common interface and connected to each other through a
broker (communication manager). The Agent Manager (AM) works as a hub through which all modules
communicate with each other. For example, issuing
a command to the speech synthesis module meansstarting voice synthesis of a given text right now “setSpeak = Now”.

3.2

Integration of Audio-Visual Information

Figure 3: Computer-graphical animated agent whosebody is combined with the Galatea face animation.
the agent system has face and body both animated bycommands received from the agent manager (AM).
Extending the above idea, we can connect information inputs and control outputs with the same agent
architecture and operate the whole system utilizing
the new input and output. In this sense, anthropomorphic spoken-dialog agent is one of excellent platforms for integrating audio-visual information fromsensors and actuators. For example, visual information may provide the user’s information of position,face expression, and gesture. Sound-source separation will be helpful not only in speech recognition innoise, but also in face/body direction control. Combined with mechanical robots, intelligent agent will
move and work physically in the real world and communicate with human users by speech, gestures andface exressions.

4

Acknowledgement

Authors would like to thank Shin-ichi Kawamoto(JAIST) for providing matrials for this article, andother colleagues of the Galatea Project, Yasuharu
Den (Chiba Univ), Keikichi Hirose (UT), Katsunobu
Itou (Nagoya U), Atsuhiko Kai (Shizuoka U), Takao
Kobayashi (TIT), Akinobu Lee (NAIST), Nobuaki
Minematsu (UT), Shigeo Morishima (Seikei U),
Satoshi Nakamura (ATR), Tsuneo Nitta (TUT), Hiroshi Shimodaira (JAIST), Keiichi Tokuda (NITech),
Takehito Utsuro (Kyoto U), Atsushi Yamada(ASTEM), and Yoichi Yamashita (Ritsumeikan U),
for their contribution to the Galatea Project. Also,
we are grateful to Masayuki Nakazawa and ChristerLunde for their additinal development (body animation) of the agent. We look forward to future collaboration with Profs. Shigeki Ando, Ayumu Mataniand Koichi Ishikawa in audio-visual information integration.

Recently, we added a body animation module to References
the Galatea Toolkit. First, we built a cartoon-type [1] Shigeki Sagayama, et al., “Galatea: An Ancomputer-graphics(CG)-based human-image animathropomorphic Spoken Dialogue Agent Toolkit,”tion (Fig. 3.2) and connected it to the Galatea agent
IPSJ Technical Report, 2002-SLP-45-10, pp. 57system. By making use of the simple Galatea archi64, Feb. 2003. (in Japanese)
tecture, connection was relatively easy. Second, we [2] http://chasen.aist-nara.ac.jp/index.html.en
combined the Galatea face and the body part of the [3] Japan Electronic Industry Development AssociaCG animation by overlapping the two animated imtion, “Standard of Symbols for Japanese Text-toages in the same window with synchronization. Now,
Speech Synthesizer,” JEIDA-62-2000, 2000.