Conversational Informatics

Conversation is everywhere

Toyoaki Nishida
Kyoto University
Conversational Informatics is a field of research aiming at
investigating human conversational behaviors as well as designingconversational artifacts that can interact with people in aconversational fashion. Based on the foundation of Artificial
Intelligence, Pattern Recognition, and Cognitive Science, we attempt
to establish a new technology consisting of environmental media,
embodied conversational agents, and management of conversationalcontents. In this talk, I will overview the field of Conversational
Informatics and highlight major results.

Goal:Artifacts that can participate in conversations
to help people create knowledge.

Conversation as interaction"show me X"

scenario A"This is X"

scenario B"This is X""hum hum""So what?"

1

Content-oriented View

Conversational Informatics
Part I: Conversational Artifacts

Conversation
SceneConversation
Scene

Conversation
ScenetEmbodied Conversational Agents
Nonverbal Expressions
Emotional Aspects
Presentation Generation in an Immersive Environment
Conversational Robots
Human-Environment
Interaction

Part II: Conversational Contents
Conversation Quantization
Integration of Natural Language Processing and Computer Vision
Video Content Acquisition and Editing
Personalization

Human-HumanInteraction

Part III: Conversational Environment Design
Human-RobotInteractionengineeringinterestscientificinterest

Ubiquitous Sensor Room for Helping Conversation
Real-time Human Proxy
Lecture Archiving System

Part IV: Conversation Measurement, Analysis and Modeling
Building and Analyzing Conversational Corpus
Quantitative Models of Nonverbal Communications
Measuring Conversation Atmosphere
Theory of Nonverbal Interaction between Humans and Agents
Model of Mutual Adaptation in Human-Robot Interaction

Part Ⅰ

Conversational Artifacts

IPOC -- Immersive Conversation Environment-- Applying Conversation Quanta
EgoChat

SPOCIPOC[Nakano, Murayama, Nishida]2Cognitive Overlapping

IPOC -- demo

Ouser
involvement

What is she
looking at?

AUOAOAUU[Okamoto, Okamoto, Nakano, Nishida][Nakano, Okamoto, Okamoto, Nishida][Nakano, Okamoto, Okamoto, Nishida]

type3

Analysis of presentation techniques on TV programs

CognitiveSympathy channel

overlap
17.9(12.8)

16.1(8.9)
42.9

type 1：{the host}

type5

type 2：{the guest}
Type 1

Type 424.8

type 3：{the subject}

16.1

type 4： {the host, the guest}

13.4

type 5： {the host, the subject}
type 6：{the guest, the subject}Type 2Type 517.9 58.2

20.4

type7

17.1

17.9

10.3 10.5

19.2 33.9

Type 7：{the host, the guest, the subject}

type1

Type 7[Nakano, Okamoto, Okamoto, Nishida]16.1 16.7

14.0(8.8)
40.4

26.3

Type 6

type6

11.9

30.8

Type 3

55.6
22.2

30.4

type4
17.9(16.1)
14.1(12.8)

type2

unit：%

Threshold:10%

3

Settings and shot generation from background photos

IPOC -- eye gaze to control the discourse[Nakano][Okamoto, Okamoto,Nakano, Nishida]

The autonomous mobile chair project

Conversational robots(1) talking(2) the robot approaches in the isual neighbor of the client(3) the client has a glance at the robot(４) approach and telling the message[Terada]4

Mobile Chair Agent -- demo

The listener robot[Terada][Ogasawara]

The listener robot -- another example[Ogasawara][Ohya, Hiramatsu]

5Part Ⅱ

Conversational Contents

Sustainable Knowledge Globe-- Accumulating Conversation Quanta
Sustainable Knowledge Globe

Contents Garden

Transforming from Other Media
S1

S2
S3

T1

T2
T0

U1
（主題連鎖）まずはこうして酢水にさらします。【個別作業】

Q1

U2（初期化）にんじんはテキストには1/2本とありますがこのまま1本使います。【食品道具提示】

U3（主題連鎖）にんじんは火の通りがいいように薄めに切ります。【個別作業】

Q2[Kubota, Nishida]Contents Garden

Reconfiguration assistance functions
distortBird’s eye viewprojection on
the cone surface

Immersive view

[Nomura, Nishida]make space

push[Nomura, Nishida]

6

Contents Garden -- demo

Manipulation by gesture-- demomagnet metaphor[Nomura, Nishida]

Manipulation by gesture-- demo[Nakamura 2005]

Text To
Presentation System

papers,reports,
books, ...

Automatic Slide
Generation

WrittenLanguage
書き言葉to話し言葉変換
Spoken Language

conversational context
Throwing[Nakamura 2005][Kurohashi]

7

This
Thisslideslideisisautomaticallyautomaticallygenerated.
generated.

Text To Presentation System -- demo
the input text
まとめ
社会問題の発生・沈滞の大きな要因として知識流通の閉塞が考えられる．その解決には，ソフトウェア・エージェント等の導入・支援が考えられるが，人間は言葉の適切さ・不適切さに対して極めて敏感であり，不十分な形式では受け入れられない．本研究はこの問題を自然言語処理技術によって解決する方策を示した．今後の課題としては，本システムをソフトウェア・エージェント等の技術と融合して会話の場に導入する実証的な実験を行い，提案した枠組みの有効性を検証する予定である．

[Kurohashi]Part Ⅲ

Conversation content acquisition
from closed caption video clips
S1

The
TheNarration
Narration[Kurohashi]

Conversation Environment Design

Acquisition from presentation with a touch panel

The ubiquitous sensor room

The meeting capture

The virtual classroom on the net

S2S3

T1

T2
T0

U1（主題連鎖）まずはこうして酢水にさらします。【個別作業】

Q1

U2（初期化）にんじんはテキストには1/2本とありますがこのまま1本使います。【食品道具提示】

U3（主題連鎖）にんじんは火の通りがいいように薄めに切ります。【個別作業】

Q2[Kurohashi]

8

Content Acquisition from Presentation with Touch Panel

Collaborative capturing of interactions bymultiple sensors

sensors in the environment(video cameras, microphones,IR trackers)

LED tags in the environment

videocameras, IRtrackers, LEDtags

microphone

communication robots
PC[Kawaguchi, Sumi]Video Contents Acquisition and Editing for Conversation Scene[Nishizaki, Ogata, Nakamura, Ohta 2002][Sumi]

Smart capture/editing/indexing[Nishizaki, Ogata, Nakamura, Ohta 2002]

9

Computer-generated virtual environment

smart capture/editing/indexing
automated capture
captureautomated indexing
object tracking / behavior recognition

viavoice992
this wayコネクタが．．．
1024
この．．．

camera control

開始：１０２４

終了：１２１９FlockofBirds

手元作業シーン
motion recognition

Real Time Human Proxy is an avatar-based interaction system, using real-time
vision-based motion sensing, human action symbolization and human action
visualization[Taniguchi, Arita]

smart capture/editing/indexing -- demo

[Nakamura]

automated editing
positionannotated data

Video Live on Demand

displayutterance
annotationPC pentiumIII
600MHz dual

[Nakamura]Computer-generated virtual environment -- demo[Taniguchi and Arita]

10

Part Ⅳ

Conversation Measurement and Analysis

Tagging the conversation
parallel annotation to verbal and nonverbal information

articulation, categorization, and structurization

Conversation corpus

camerasarticulation
categorization
yoru go h a N

structurization"night"

q te"supper" "about"articulation
categorization

head declines

structurization

head declines
nodding

Mutual adaptation

articulation
categorization y

Causal model of conversationso r

structurizationu"night"

g oha

Nqte

"about""supper"

articulation
categorization

structurization

head declineshead declines
nodding[Den]

Communication Atmosphere[Rutkowski 2004]

A causal model concerning the intention ofparticipation in a community and using a tool
透明性.21☦

the scheme

mutual information estimate

communication efficiency[Matsumura]

11

[Komatsu et al 2004]

Model of mutual adaptation

Conversation Quantization
conversion /spatialization

conversation
conversation
conversation
quantumquantum quantum

accumulation
dematerialization
conversation
quantum

The environment of experimentation

conversation
quantum

conversation
quantummaterialization

RSA>MWSA: game score increasing

RSA>MWSA: game score decreasing

RSA: Respiratory sinus arrhythmia, MWSA: Mayer wave sinus arrhythmia

Conversation Quantization (elaborated view)

embodied conversational agent

another conversationalsituation

Representing Conversation Quantum
socialintelligence
quantity

conversational situationa conversational situation

embodied
conversational
agent

conversationas a phenomenon

Representation of a Conversation Quantum
Representation of Content

measure"This tree is called a"Kai no ki". It wasplanted forcommemorating
Professor Koshiba's
Nobel Prize inPhysics "information landscapeconversation quantum
conversation quantum
conversation quantum

additional services
such as navigation,summarization, ...
Representation of Interaction…
《introduction》

capturetemporal
expansion

spatialexpansion

conversion

archives《focus attention》《add new information》

editor…

12Representing Conversation Quantum
conversation
as a phenomenon

Robot as an embodied knowledge medium

Representation of a Conversation Quantum
Representation of Content"This tree is called a"Kai no ki". It wasplanted forcommemorating
Professor Koshiba's
Nobel Prize inPhysics "Listener Robot

conversation quantum
conversation quantum
conversation quantum

Presenter Robot

IPOC

Representation of Interaction…
《introduction》《focus attention》《add new information》…

Ohya, Hiramatsu

Dynamic binding of Content and Interaction
ScenesConversation QuantaContent

Difference from previous work
previous work

this worktask-oriented dialogue

conversation as content

dialogue as a means
verbal communication
rule-based methods

conversation as the goal
multimodal communication
data-driven methods"attach the rear wheel"Interaction…

Content《show it to the listener》"open the lever andpush the wheel"《speak to the listener》…

Interaction…
《focus attention》《speak to the listener whileattaching the wheel》…

focus of the discourse − objects/events focused in interaction
part/whole − semantic gesture suggesting the hierarchy
key objects − those that are demonstrated with emphasis
key action/movement − those that encourage the listener synchronize
same meaning − those displayed by the same gestural features

13Analysis, Support, and Materialization of Real World Interaction

Directions

Toyoaki Nishida, Yuichi Nakamura, Kazuhiro Ueda, Sadao Kurohashi, Yasuyuki Sumi, Igor Pandzic

Real World Interaction
Content

Real World Interaction
CorpusIntelligence → Communication

utilization ascontent

automatedarchiving

content
production

content utilization

Research Agenda

Real World Interaction
Data

Algorithms → Phenomena

media data（image, speech, sound）
motion data (kinematics, dynamics)
physiological data（gaze, blood, perspiration,tactile sense）

real world knowledge sharing(beyond the scope of this research)

Interaction Content Acquisition and Utilization

display
controlling robots- automated acquisition using commonsense knowledge- media conversion, materialization, embodimentInteraction Science- automated acquisition using commonsense knowledge- grounded corpus
augmented conversation environment
augmented conversation environment

human-environment
human-environment
interaction
interaction
virtual assistantvirtual assistantInterface → Contents

real world assistantreal world assistanthuman-human
human-humaninteraction
interaction

human-agent
human-agent
interaction
interaction

Interaction Support Technology- natural communication without disturbance- mutual adaptation

collaboration with・Interaction Measurement and Analysis Environment
・Super Precision Computer Vision
・Humanoid Robotics

Summary - Conversational Informatics
Part I: Conversational Artifacts
Embodied Conversational Agents
Nonverbal Expressions
Emotional Aspects
Presentation Generation in an Immersive Environment
Conversational Robots
Human-Environment
Interaction

Part II: Conversational Contents
Conversation Quantization
Integration of Natural Language Processing and Computer Vision
Video Content Acquisition and Editing
Personalization

Human-HumanInteraction

Part III: Conversational Environment Design
Human-Robot
Interactionengineeringinterestscientificinterest

Ubiquitous Sensor Room for Helping Conversation
Real-time Human ProxyLecture Archiving System

Part IV: Conversation Measurement, Analysis and Modeling
Building and Analyzing Conversational Corpus
Quantitative Models of Nonverbal Communications
Measuring Conversation Atmosphere
Theory of Nonverbal Interaction between Humans and Agents
Model of Mutual Adaptation in Human-Robot Interaction

14