ISSN 2186-7437

NII Shonan Meeting Report

No. 2015-7

The Future of Human-Robot Spoken
Dialogue:from Information Services to Virtual
Assistants
NII Shonan Meeting Report
Rafael E. Banchs
Sakriani Sakti
Etsuo Mizukami
March 26–28, 2015

National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda-Ku, Tokyo, Japan

The Future of Human-Robot Spoken Dialogue:
from Information Services to Virtual Assistants
NII Shonan Meeting Report
Organizers:
Rafael E. Banchs (Institute for Infocomm Research, Singapore)
Sakriani Sakti (Nara Institute of Science and Technology, Japan)
Etsuo Mizukami (National Institute of Information andCommunications Technology, Japan)
March 26–28, 2015

Meeting Motivation and Overview
Dialogue Systems embrace the ultimate goal of human-robot interaction, in
which computational systems communicate with their human interlocutors inthe same way humans communicate among them. Although significant progresshas been achieved during the last few years and some pioneering commercial
systems are already finding their way to the market, current state-of-the-art
dialogue systems are very limited in their approach to the human communication
phenomenon. Some of these limitations include:• the lack of ability to properly model world knowledge for reasoning purposes,• the still low reliability of speech recognition and the limited capacity of
dialogue systems to cope with recognition errors in a logical manner,• the complex and multimodal nature of the pragmatic phenomena and thesocio-cultural aspects that aﬀect such interactions, and• the diﬃculty of evaluating dialogue quality and the subsequent development of reliable strategies for automatic learning and adaptation.
The main objective of this NII-Shonan meeting was to discuss about themost relevant and promising future directions of research in dialogue systems.
The discussion was centered on how these research directions address the diﬀerent problems and limitations of current dialogue systems, as well as how they
provide the basis for the next generation of intelligent artificial agents.
The participants were requested to present work in progress as well as topropose new collaborative eﬀorts to tackle the main limitations of current dialogue systems mentioned above. As a result of the meeting, a research agenda
for main directions and international collaboration for the next three to five
years was defined.
1

Joint organization of future workshops and shared tasks by the participantsin the meeting were defined with the objective of pushing the state-of-the-art indialogue systems to a more comprehensive eﬀort for developing a new generationof intelligent virtual assistants.

Seminar Schedule
March 25th (Wednesday): Check-in
15:00- Check-in
19:00- Welcome Banquet

March 26th (Thursday): Day 1
208)

Full-day session (Room

07:30-08:30 Breakfast
08:30-08:35 Seminar Opening
08:35-10:00 Session 1: Social Human-Robot Interaction
10:00-10:30 Break
10:30-12:00 Session 2: Artificial Conversational Agents
12:00-13:30 Lunch
13:30-15:00 Session 3: Knowledge Representation and Dialog Modelling
15:00-15:30 Break
15:30-17:00 Session 4: Spoken Dialog and Paralinguistics
17:00-18:00 Free discussion period
18:00-19:30 DinnerMarch 27th (Friday): Day 2 Full-day session (Room 208& 209)
07:30-08:30 Breakfast
08:30-09:00 Team Formation and Round Table Instructions (Room 208)2

09:00-10:00 Session 5: Round Table Discussions (Room 209)
10:00-10:30 Break
10:30-12:00 Session 6: Round Table Discussions (continuation)
12:00-13:30 Lunch
13:30-15:00 Session 7: Plenary Sessions (Part I) (Room 208)
15:00-15:30 Break
15:30-17:00 Session 8: Plenary Sessions (Part II)
17:00-18:00 Free discussion period
18:00-20:00 Main Banquet

March 28th (Saturday): Day 3
208)

Half-day session (Room

07:00-10:00 Check out
07:30-08:30 Breakfast
08:30-09:00 Working Committee Formation
09:00-10:00 Session 9: Working Committee discussions
10:00-10:30 Break
10:30-12:00 Session 10: Road map presentations for selected initiatives
12:00-13:30 Lunch
13:30 Dismiss

Overview of Talks in Session 1:
Social Human-Robot Interaction
Session Chair: Satoshi Nakamura
Symbiotic human robot interaction
Tatsuya Kawahara, Kyoto University
A new project on symbiotic human robot interaction is introduced. The goal3

of the project is autonomous humanoid robot who looks like human, behaves
like human, and interacts like human. Key technologies include robust speech
recognition, dialogue modelling and multi-modal interaction modelling.

Engagement and natural interaction in human-robot interactions
Kristiina Jokinen, University of Helsinki
In human-human communication, a wide repertoire of multimodal signals isused to provide eﬀective feedback about the parther’s interest and understanding. In a similar manner, multimodality is important in the context of “social
robotics”, where the robot is meant to support human users in social, interactive
tasks, and to serve as an intuitive interface to access and share information.
The interlocutors’ experience of the interaction is largely based on their holistic interpretation of the partner’s multimodal behaviour, besides the content of
the verbal communication. We hypothesise that by observing the user’s behaviour when interacting with a humanoid robot, it is possible to estimate how
a shared understanding is constructed through interaction among the interlocutors, and consequently, to predict the users experience of their interaction withthe robot. The users assessment of the interaction (responsiveness, expressiveness, interface, usability, overall impression) indeed seems to correlate with theirmultimodal behaviour, i.e. the interlocutors engagement and active participation relate to their assessment of the success of communication. This allows us
to establish measurements for the users engagement and dialogue management
in robot interactions, and points towards automating evaluation of human-robot
interactive situations.

Appropriateness of behaviours as a participant of conversations: can a robot participate in the multi-party dialogue?
Etsuo Mizukami, National Institute of Information and Communications Technology
In the near or long-term future, a robot may exist as a member of our society.
What a robot should do to be such an existence even in the conversation made bymulti-party? In order to answer this question, we have to know how we humanbehave in the multi-party dialogue. Although a robot need not to behave assame as a human, it is required at least to understand some norms or rules ofhuman interaction. For example, it has to know when it should start to speak
relevant words as a proper role, and what it should do as an appropriate hearer.

Human-robot interaction evaluation and ethics
Joseph Mariani, LIMSI-CNRS & IMMI
I coordinated within the CNRS Ethics Committee (Comets) a working group
who produced a report on the Ethics of research in ICT. This report contains
several recommendations. Generally speaking, the report expressed the factthat in order to be more respectful of ethics, research in ICT should include4

a reflection on the consequences of its results at the time when the researchis being conducted, and eventually adapt or complete the scope of research.
It went with several examples, including one on robotics: one of Asimov laws
mentions that a robot shouldnt harm a human. This is only possible if therobot is able to identify humans, and make the distinction between humans andobjects, animals or other robots. It therefore implies that the robot possesses
vision abilities, which is the scientific field of Computer Vision. The present
state-of-the-art in Computer Vision appears however insuﬃcient to guarantee
an acceptable categorization by the robot between humans and non-humans.
It is therefore necessary to invest on Computer Vision in order to measure thequality of the systems, the progress in quality and the adequacy with the robot
needs in order to behave in an ethical way.
The report proposed to launch a specific national Ethics Committee for ICT
research, which is now active. The first report of this committee deals with the
ethics of research in robotics, and extends very largely the conclusions of thefirst report. One of the issues that it contains concerns the traceability of therobots actions, and the fact that it should be able to explain clearly and quickly,
and make understandable, the reasons of its decisions and actions, based on its
perception.
Together with Laurence Devillers who participated in this report on researchin robotics, I would like to propose to carry on a reflection within this meeting on
the evaluation of human-robot interaction, as a necessary tool in order to ensure
the ethical behaviour of robots. This would include the study of the various
components and functions that should be evaluated, of the general organizationof evaluation, of the various metrics that could be used, either based on humanor automatic evaluation, of the production and distribution of development andtest data, including the questions of privacy, etc.

Aﬀective and social robotics: engagement, evaluation andethicsLaurence Y. Devillers, LIMSI-CNRS/Univ. Paris-Sorbonne
I would like to propose to carry on a reflection within the Shonan meetingon advanced dialogues employing complex social behaviours in order to provide
a companion-robot with the skills to create and maintain a long term social
relationship through verbal and non verbal language interaction. Talk duringsocial interactions naturally involves the exchange of propositional content but
also and perhaps more importantly the expression of interpersonal relationships,
as well as displays of emotion, aﬀect, interest, etc. Such social interaction requires that the robot has the ability to represent and understand some complex
human social behaviour. Cognitive decisions will be used for reasoning on thestrategy of the dialog and deciding more complex social behaviours (humour,compassion, white lies, etc.) taking into account the user profile and contextual
information. Social interactions require social intelligence, cultural understanding and robust multimodal perception and employ theory of mind for inferring
the cognitive states of another person. It is not straightforward to design arobot with such abilities and it is necessary to combine pluri-disciplinary skills
and theories.

5

This reflection will also include the evaluation of such system and the various
metrics that could be used like the measure of social engagement with the user.
With Joseph Mariani we would like to carry out a discussion on evaluation andalso on how researchers could incorporate ethical constraints at the stages ofdefining and implementing their research projects.

Models of culture-specific dialogue
David Traum, USC ICT
Virtual humans are artificial agents that include both a visual human-likebody and intelligent cognition driving action of the body, including engaging inface to face conversation. Culture covers a wide range of common knowledge of
behaviour and communication that can be used in a number of ways includinginterpreting the meaning of action, establishing identity, expressing meaning,
and inference about the performer. Virtual human and robot behaviour will
always be interpreted by people from a culture-specific vantage point and viewers
will make inferences about cultural aspects of the virtual humans, so whether
or not an explicit model of culture is used in the design and behaviour of thevirtual humans, one will be attributed to them. In this talk, I will present ataxonomy of types of culture models for virtual humans and look at several
examples of existing cultural models that have been used, focusing primarily on
those we have developed at the Institute for Creative Technologies at Universityof Southern California, and point out remaining steps for a more full model ofculture.

Overview of Talks in Session 2:
Artificial Conversational Agents
Session Chair: Wolfgang Minker
Multiparty and open domain reasoning dialog system
Gary Geunbae Lee, POSTECHWill present recent postech approaches for multi-party dialog system andintelligent assistant dialog system which shows almost open domain language
understanding and knowledge based reasoning capabilities.

Turn-taking and attention in human-robot dialogue
Gabriel Skantze, KTH
At KTH we have been doing research on multi-modal dialogue systems fora long time, often based on observations of human-human dialogue. In recent
years, we have focused on issues related to human-robot dialogue, such as situated interaction, multi-party dialogue and visual attention. We have developed
several tools for conducting this research, including the dialogue system framework IrisTK (released as open source) and the back-projected robot head Furhat(which is now a commercial product). I will describe how we have used these6tools for studying how turn-taking and attention is coordinated in situated,
multi-party dialogue. This includes controlled experiments in the lab, as wellas large scale field trials in public settings.

Towards open-domain conversational systems
Ryuichiro Higashinaka, NTT
Although task-oriented dialogue systems have been actively investigated in
the past decades, it is only recently that open-domain conversational systems,
or chat systems, have been attracting attention. Such systems need to handle open-domain utterances, which poses many challenges in dialogue systems
research. In my presentation, I describe our ongoing work on using predicate
argument structures (with zero-anaphora resolution) for open-domain utterance
understanding and using social media as resources for natural language generation about a wide variety of topics. I also talk about how to personalize suchopen-domain conversational systems for long-term use. I briefly describe theproject in Japan which aims at detecting dialogue breakdown in open-domain
conversation.

A chat agent based on the vector space model
Rafael E Banchs, Institute for Infocomm ResearchThis presentation will focus on IRIS (Informal Response Interactive System)a chat engine that implements a vector space search approach over a data collection of movie dialogs. I will describe in detail the basic principles of operation of
the system, along with the added capabilities of vocabulary learning and styleand manner adaptation. A comparative evaluation between the proposed system and a baseline system is also presented. Finally, future plans of research inthis area will be shared.

Developing conversational agents and the studies of aﬀective and cognitive processes
Sakriani Sakti, Nara Institute of Science and Technology
Interaction between human and computer continues to change to the better replicate interaction between humans. The aim is to build a conversational
agent that can interact with human in as natural a fashion as possible. In thistalk, I will present some ongoing research works in developing conversational
agents. This includes chat-based dialog management, emotion recognition, paralinguistic expression, and cognitive communication.

CLARA: a virtual agent for conference information andtouristic assistance
Haizhou Li, Institute for Infocomm Research
I will present some results from our recent experience on deploying a virtual
agent called CLARA at INTERSPEECH 2014. The virtual agent provided con7

ference delegates with information related to the conference’s technical program
and related events, as well as touristic information about Singapore.

Overview of Talks in Session 3:
Knowledge Representation and Dialog Modelling
Session Chair: Gary Geunbae Lee
Incorporating knowledge into word representation learning
Kevin Duh, Nara Institute of Science and Technology
Recent work has shown that word vector representations, also known as
word embeddings, can successfully capture semantic and syntactic regularities
in language (Mikolov2013) and improve the performance of various Natural
Language Processing systems (Collobert2011). While many methods have been
proposed, most are based on the same premise of ”distributional semantics”,
where words from similar contexts are mapped to nearby vectors.
However, distributional semantics is by no means the only way to model
word meaning. In this talk, I will discuss our attempts to incorporate relationalsemantics and world knowledge into word representation learning algorithms.
I will describe a general algorithm based on the Alternating Direction Methodof Multipliers (ADMM) that enables us to integrate flexible knowledge sources
such as WordNet and FreeBase.

Towards improving virtual assistants intelligence
Jiang Ridong, Institute for Infocomm Research
Speech interface is an intuitive, flexible and natural means of communication
between users and machines. With the advancement of natural language processing technology, more and more voice enabled applications and products are
emerging in our daily life. One of the great challenges is how to improve the intelligence of these virtual assistants. Various systems with diﬀerent intelligences
have been developed by researchers from all around the world. However, there
is still a long way for a machine to pass the Turing test to show its intelligence.
This requires long-standing eﬀorts of the research community on this topic.

Knowledge based AI for service robotics
Suraj Nair, TUM Create
In order to systematically introduce robots into human spaces, the usabilityof such robots needs very close attention. Human with no knowledge in robotics
should be able to command and interact with such systems in a intuitive andergonomic manner. The main challenge here is to build system which is capableof receiving under-specified instruction from a human and still be able to executethe tasks successfully. In order to achieve this we propose a Knowledge Basedsystem where domain knowledge is expressed in a semantic form. The challenge
here is to model common sense knowledge and logical reasoning tools which can8infer missing pieces of information important for task execution. We will discuss
our eﬀorts in this direction and current results.

Dialog state tracking on human-human dialogs
Seokhwan Kim, Institute for Infocomm Research
Dialog state tracking is one of the key sub-tasks of dialog management, which
defines the representation of dialog states and updates them at each moment
on a given on-going conversation. Most previous work on dialog state tracking
has focused on developing trackers based on given data which consist of goaloriented human-machine conversations for searching bus schedules, restaurant,
or tourist information. Alternatively, we suggest a new dialog state tracking
task on human-human dialogs as a shared task. Although it would be diﬃcult
to apply the outputs from this task immediately to any practical systems, we
expect this as a first step to develop much more human-like systems in a longterm point of view.

WFSD based multilingual spoken dialogue management forhuman-robot communication
Takuma Okamoto, National Institute of Information and Communications Technology
For realizing human-robot spoken dialogue systems, not only one user situation but also multiuser situation should be considered. In addition, in globalizedsociety, multilingual spoken dialogue systems are required so that a robot canspeak multilingually for multinational uses simultaneously. The WFST based
spoken dialogue manager can realize these multilingual spoken dialogue systems
by unifying each spoken language understanding to all together as a multilingual
spoken language understanding WFST.

Dialog control considering systems goal and incongruity detection by brain signals
Satoshi Nakamura, Nara Institute of Science and Technology
My presentation includes 1) the current dialog paradigm needs be moregeneralized so that the system can persuade users or negotiate with users, 2)
the dialog control needs to consider paralinguistic information like emotions, 3)
realtime objective evaluation methods needs to be developed, and 4) the persona
for the dialog system needs to be considered.

9Overview of Talks in Session 4:
Spoken Dialog and Paralinguistics
Session Chair: Kristiina Jokinen
Conversational interfaces; issues and technologies
Michael McTear, University of Ulster
My current research is in the area of conversational interfaces. We are looking at how people will be able to communicate with devices such as smartphones,
robots, smart watches, and other wearables. A colleague and I are currently
writing a book on this topic and one of our main tasks is to define the issues
involved and technologies required to address them. One of the main challenges
concerns the interpretation of the users input and the determination of the devices response. Identifying the users intent involves a combination of dialogue
act recognition and topic identification for example, determining whether theuser is requesting information, some action, or just making conversation. Given
that devices are connected and that they can provide useful information about
the conversational context, the user location, as well as information provided
by biosignals, the context for interpretation is much wider than in traditional
spoken language understanding. Next the system has to decide the appropriate
action to take for example, an Internet search, a physical action, a conversational turn and which device or service is involved. Finally the response has tobe formulated, for example, as a combination of speech and action, a summaryfrom a web document, or a multimodal output. Although these issues have been
investigated extensively in the diﬀerent technologies that contribute to spoken
dialogue systems - spoken language understanding, dialogue management, andresponse generation -, their application in the area of conversational interfaces
to multiple connected devices provides many new challenges and opportunities.

Interaction with cognitive technical systems
Wolfgang Minker, Ulm University
Spoken Language Dialogue Systems (SLDSs) providing natural interfaces tocomputer-based applications have been an active research area for many years.
However, most state-of-the-art systems are still static in use, and their fieldof application is rather limited. Our work aims at overcoming this limitation.
In this presentation, we will provide a brief overview on our research activities
in the domain of adaptive and assistive SLDSs for next-generation Cognitive
Technical Systems. These activities include Emotion Detection from Multimodal Signals, Verbal Intelligence Estimation based on the analysis of Spoken
Utterances and Statistical Modelling for User-centred Adaptive Spoken Dialogue
Systems.

Spoken dialog technologies in NICT
Atsuo Hiroe, National Institute of Information and Communications Technology
This presentation introduces spoken dialog technologies developed in Na10

tional Institute of Information and Communications Technology, NICT. One ofissues in a spoken dialog system is the trade-oﬀ between accuracy of its response
and portability of language or domain. In order to address this issue, we represent both a model for language understanding and a dialog scenario as a finite
state transducer (FST) and drive it with an FST decoder. This framework caneases the trade-oﬀ since FST has potential to express both a rule and a statistical model while FST decoder is language and domain independent. We also
introduce actual porting works.

Flexible speech generation and input technologies for making man-machine communication more eﬀective
Tomoki Toda, Nara Institute of Science and Technology
Speech can convey not only linguistic information but also para-linguistic information and non-linguistic information simultaneously. To make man-machinecommunication more eﬀective, it is useful to develop technologies for handlingpara-/non-linguistic information as well, e.g., a speech generation technique tomake it possible to flexibly control para-/non-linguistic information. Moreover,it will also helpful for making man-machine communication available anytime
and anywhere to develop technologies for achieving more flexible speech input,
such as a silent speech technique for privacy purpose. In this talk, I will introduce our recent work on the development of flexible speech generation andinput technologies.

Help your robot understand you: cooperative learning incomputational paralinguistics
Björn Schuller, University of Passau & Imperial College London
Human-Robot Spoken Dialogues to the present day mostly lack emotionaland social intelligence. This can be overcome to some degree by lending our new
electronic companions the ability to analyse our state and trait in a reliable androbust manner. In this light, the state-of-the-art in computational paralinguistic
speech analysis for robots is highlighted based on the Interspeech ComParE
2009-2015 series initiated and co-organised by the presenter. To improve on
these results, avenues towards largely autonomous learning abilities of a robot
without the need to permanently require ”more input” from the user will be
discussed. To this end, the novel cooperative learning approach is presented as
an utmost eﬃcient blend of semi-supervised and active learning. The required
confidence measurement is quickly introduced as well. The talk is rounded up
by a more general view on current activities of the speakers two groups at the
University of Passau/Germany and the Imperial College London/UK in thisfield including big-data deep recurrent memory-enhanced multi-task learning,transfer learning, and distributed processing.

Paralinguistic processing for dialogue systems
Nick Campbell, Trinity College Dublin, The University of Dublin
Speech recognition and synthesis are now mature technologies but are still11

predominantly text-based; they are not suﬃciently tuned to the characteristics
of spoken language. In particular, they typically lack the ability to process thesocial signals that accompany propositional content in spoken interactions. At
TCD we have been building dialogue systems that present solutions to this problem; the incorporate non-verbal information (both speech and image) alongside
more traditional dialogue information and have been testing these devices inrobotics, call-centre, teaching, and elderly-care applications.

Overview of Round Tables and Working Committee Discussions
During the second and third day of the meeting, Round Table and Working
Committee discussions were conducted respectively. The five selected topics
and their corresponding Round Table and Working Committee members wereas follows:• Group 1: Multimodality and Social Interaction
Participants: Tatsuya Kawahara, Kristiina Jokinen, Laurence Devillers,
Etsuo Mizukami, Gabriel Skantze, Gary Geunbae Lee• Group 2: Automatic Speech Recognition, Natural Language Understanding and Natural Language Generation
Participants: Michael McTear, Seokhwan Kim, Atsuo Hiroe, Björn Schuller,
Tomoki Toda
• Group 3: Knowledge Representation and Reasoning
Participants: Kevin Duh, Jiang Ridong, Suraj Nair, Sakriani Sakti, Haizhou
Li• Group 4: Dialogue Management Paradigms and Policy Learning
Participants: Satoshi Nakamura, Nick Campbell, Takuma Okamoto, David
Traum• Group 5: Automatic Evaluation and Resources
Participants: Ryuichiro Higashinaka, Joseph Mariani, Wolfgang Minker,
Rafael Banchs
The main objective of Round Table discussions was to focus on specific topics
to identify the main challenges currently faced by research work in each areaand define specific actions (i.e. workshops, exploratory project, shared task orcompetitions, open source development, etc.) to be proposed to the community.
The main objective of the Working Committee discussions was to select specificactions proposed during the previous day and generate: an actiona plan for theselected action, a tentative timeline of activities, and a list of required resources
and potentially interested teams.

List of Participants and Aﬃliations
The following is a complete list of participants (listed in no particular order)
and their aﬃliations:12• Prof. Satoshi Nakamura, Nara Institute of Science and Technology, Japan• Prof. Tatsuya Kawahara, Kyoto University, Japan• Dr. Ryuichiro Higashinaka, NTT, Japan• Prof. Gary Geunbae Lee, Postech, South Korea• Dr. Haizhou Li, Institute for Infocomm Research, Singapore• Dr. Suraj Nair, TUM Create Singapore / TUM Germany, Germany• Prof. Wolfgang Minker, Ulm University Communications Engineering,
Germany• Prof. Michael McTear, University of Ulster, UK• Prof. Nick Campbell, Trinity College, Dublin, Irland• Prof. Kristiina Jokinen University of Helsinki / University of Tartu, Finland• Prof. Joseph Mariani, LIMSI-CNRS & IMMI, France• Dr. David Traum, USC ICT, USA• Prof. Laurence Devillers, LIMSI-CNRS/University Paris-Sorbonne, France• Assoc. Prof. Tomoki Toda, Nara Institute of Science and Technology,
Japan• Assis. Prof. Kevin Duh, Nara Institute of Science and Technology, Japan• Dr. Seokhwan Kim, Institute for Infocomm Research, Singapore• Dr. Ridong Jiang, Institute for Infocomm Research, Singapore• Prof. Björn Schuller, Imperial College London / University of Passau, UK• Assis. Prof. Gabriel Skantze, KTH Royal Institute of Technology, Sweden• Dr. Takuma Okamoto, National Institute of Information and Communications Technology, Japan• Mr. Atsuo Hiroe, National Institute of Information and Communications
Technology, Japan• Dr. Rafael E. Banchs, Institute for Infocomm Research (I2R), Singapore• Assis. Prof. Sakriani Sakti, Nara Institute of Science and Technology(NAIST), Japan• Dr. Etsuo Mizukami, National Institute of Information and Communications Technology (NICT), Japan

13