Augmented Human Communication
Laboratory
Prof. Satoshi Nakamura
Graduate School of Information Science,
NARA INSTITUTE OF SCIENCEAND TECHNOLOGY
Prof. Satoshi Nakamura @AHC Lab IS NAISTAugmented Human Communication Lab
What is Augmented Human
Communication?
Communication
Empowerment
Technologies

Voice
Conversion

Intelligent
Honnyaku
Translation
Robot
Konnyaku
Robot
Prof. Satoshi Nakamura @AHC Lab IS NAISTC3PO

Talking
VehicleBrain
Communication

Research for the Future
Brain
Analysis
Dr. Nakamura
Dr. Toda

Persona
Modeling

Dr. Sakti

Speech Translation

Dr. Neubig

Big DataAnalysis
Ms. Matsuda

Why don’t you
join our lab!

Voice QoL

SpokenDialogVoice Conversion
Speech Synthesis

Integrating fundamental technologies into
the augmented human-communication systems

SRG(Super Research Group): New framework of NAIST forwider collaboration inside/outside the lab.
Prof. Satoshi Nakamura @AHC Lab IS NAISTWorld-wide
Speech Translation
Speech Recognition
Dialog Control

Speech Analysis,Conversion, Synthesis

Cognitive Communication

Speech Signal Processing

Big Data Analysis

Dialog Generation
Voice QoL
（Voice Bank）

Karlsruhe Instituteof TechnologyProf.
Satoshi Nakamura
Cambrige
University

ATR, NICT

Assoc. Prof
Tomoki Toda

Carnegie Melon
UniversityUniversity of Ulm
University of Ilinois

Speech Recognition
Multilingual SRRutgers, NewJersy
State University

Cognitive Communication

Machine Translation
Speech Translation

Graphical Models

Natural Language Processing

Machine Learning
Assist. Prof.
Sariani Sakti

Bandung Instituteof Technology

Assist. Prof.
Graham Neubig

Prof. Satoshi Nakamura @AHC Lab IS NAISTMembers

D: 3, M2: 9,
KIT Students: 2

Spoken Dialog
M: 2

SpeechTranslation
D: 1,M: 3

Cognitive
Communication
D:1, M:1

SpeechSynthesis
D:1, M:3

Prof. Satoshi Nakamura @AHC Lab IS NAISTNew Members from Apr. 2012

New 11 M1students

Speech TranslationSpeech Translation System
Nice to meet you!

はじめまして！
SpeechRecogni‐tion (ASR)

MachineTransla‐tion (MT)SpeechSynthesis(SS)History of Speech Translation Research In Japan
1986

1992

NAIST

1999

2006

2008

ATR

ATR

NICT

ReadSpeech

Daily
Conversation

Wider andReal Domain+ More languages
for translation• Syntactically correct• Clear utterance• Limited domain
Ex. “ConferenceRegistration”• Standard expression• Unclear utterance• Limited domain
Ex. “Hotel Reservation”

ATR•21 multilateral text translation
VoiceTra• Wider and real domain“International Travel”
A-STAR• Realistic expressions• Noisy speech• J-E, J-C speech translation • Multilateral translation for 8 Asian languages• Network-based S2ST

C-STAR

Fundamentals

IWSLT• Multilateral translation for 7 world languages

Rule-based Technology
Hand-made20102011• Evaluation Campaign of S2S technologies

Corpus-based Technology

Large scale corpus+ Machine learningSpeech Translation
Speech Translation System
Nice to meet you!

はじめまして！
SpeechRecogni‐tion (ASR)

Optimization

MachineTransla‐tion (MT)SpeechSynthesis(SS)

Delay

Naturalness

Optimization of ASR

Simultaneous Translation

Speech to speech

Optimization of ASRconsidering MTpeformance

Low delay speechtranslation for Japanese
to English
translation

Richer expressive
speech synthesis forspeech translation

Prof. Satoshi Nakamura @AHC Lab IS NAISTSpeech Synthesis
Voice Conversion
Convert voice to any speaker andvoice quality and voice conversion
for speech-to-speech translation.
HelloHello

Input speaker

Target Speaker

Text-to-Speech Synthesis
Speech synthesis from text

“Thank you”

Thank you

Synthesis with emotions,individuality, quality,communicability.

Voice QoL

Silent Speech Interface

For vocally handicappedpeople who cannot speak in ausual way...
Recover their natural voices!
Convert their artificiallygenerated voices into morenaturally sounding voices

Communication without disturbing
surrounding people by non-audible
speech input？

Card No.is○○○!

Card No. is○○よ？

？

Prof. Satoshi Nakamura @AHC Lab IS NAISTPersuasive Dialog System
Estimate user interests and persuade users
to the target topics and interests
Target Topics
Suitable labs
to me?

Want tojoin AHClab！

Conditions for persuation
Want user tojoin AHC lab.

DialogPersuation
OK!

・Find users interests
・Persuade user to be interested totarget topics through dialogs

Estimate Interests

Persuasive Dialog

Estimateinterests byuser utterances

Dialog control bypersuasive responses
to target topics
and interests

2011©Yamauchi AHC‐Lab, IS, NAIST

Cognitive Communication
Automatic Communication Measurements•

Detection of
cognitive andsemantic
mismatch

Objective communication
measurement from EEG

Social Communication Skill Training by
iPad

Gaze tracking
during conversation
Prof. Satoshi Nakamura @AHC Lab IS NAISTInterACT

Karlsruhe
Instituteof Technology

WasedaUniversity

Carnegie Melon
University

NationalInstitute of ICT

Italian Instituteof Technology

Hong Kong
University ofScience andTechnology
Nara Institute ofScience andTechnologyUniversity SouthernCalifornia

Joint Research Project, Student Exchange, Faculty ExchangeProf. Satoshi Nakamura @AHC Lab IS NAISTProfessor Satoshi Nakamura
Background 1981.4- 1994.3 Sharp Corp. Central Research Labs
 1986-1989 ATR Interpreting Telephony Res. Labs.

Spoken Language
Communication
ResearchLaboratories

 1994.4-2000.3 Associate Prof. Nara Institute of Science and Technolog
 2000.4 Advanced Telecommunication Research International (ATR) Vice President of ATR, Director of Spoken Language Communication Labs.
 ATR Fellow

 2006.4

National Institute of Information and Communication Tech.(NICT) Director, MASTAR Project Director, KCCC Research Center Director General, Keihannna Research Laboratories
 Dec. 2003 Honorarprofessor of University Karlsruhe, Germany Apr. 2011 Prof. at Nara Institute of Science and Technology/14 14

15Speech to Speech Translation
Launched in November 2007
The first network‐based STStranslation service

・“VoiceTra” Network-based Speech Translation
released on Jul. 2010
・21 language pair for Text I/O

音声翻訳 「しゃべって翻訳」

500k download and 4M access worldwide so far.

・6 language pair for Speech I/O

・日英双方向・NTTドコモトップの画面

音声入力画面

翻訳結果出力画面

VoiceTra

Japanese, English, Mandarin, Taiwanese Mandarin,
German, French, Dutch, Danish,Italian, Spanish, Portuguese, Brazilian Portuguese,Russian, Arabic, Hindi, Indonesian, Malay, Thai, Tagalog,Vietnamese, Korean※ Language in red can be input/output in voices.
15※There is no text input support for Hindi or Vietnamese.
/14

Associate Professor
Tomoki Toda
Education
1999.4 Graduate School of Information Science, NAIST- Master degree in engineering, 2001.3- Doctor degree in engineering, 2003.3

Professional Experience
2003.4 JSPS Research Fellow (ATR, CMU, NITECH)
2005.4 NAIST Assistant Professor
2011.4 NAIST Associate Professor

AwardIEEE Signal Processing Society
The 2009 Young Author Best Paper Award
Nippon Ericsson K.K.
The 10th Ericsson Young Scientist Award

Assistant Professor Sakriani SaktiEducation
2005-2008

2000-2002

1995-1999

Work Experience
2011 – Now
2009 – 2011
2006 – 20112003 – 2009
2001-2002
1999-2000

Doctorate degree (Dr.-Ing)
in Engineering Science,
University of Ulm,
GERMANY
Master degree (MSc )in Communication Technology,
University of Ulm, GERMANY
Bachelor degree (BSc)
in Informatics, Bandung
Institute of Technology, INDONESIA

Assistant Professor, Augmented Human Communication Labs, NAIST, JAPANVisiting Professor, Faculty of Computer Science, University of Indonesia, INDONESIA
Expert Researcher, Spoken Language Communication Research Groups, NICT, JAPANResearch Engineer - Researcher, Spoken Language Communication Research Labs, ATR, JAPANMasterarbeit, Speech Understanding Dept,Daimler Chrysler Research Center, GERMANY
Junior Software Consultant, Sumarno Pabotingi
Associate, INDONESIA/14 17

Assistant Professor Graham Neubig
Background2001-2005 University of Illinois, Urbana-Champaign
B.E. Computer Science
2005-2008 Worked as English Teacher/Translator
2008-2012 Kyoto University, Doctor in Informatics
2012Assistant Professor, NAIST

Machine Translation

Spoken Language
Processing

奈良先端へようこそ！

Learning from Speech

Welcome to NAIST!
Character-based MT
奈良this isa pen
Speech Transformation

raThat’s, um, real good

Nara

That’s really goodna

Natural Language
Analysis
言葉を扱う
Segmentation

言葉を扱う
Pronunciation

kotoba oatsukau

Part-of-SpeechNPV… and many more!