1Semi-supervised Learning by Machine Speech Chain for
Multilingual Speech Processing, and Recent Progresson Automatic Speech Interpretation
Satoshi Nakamura,
Sakriani Sakti, and Katsuhito SudohGraduate School of Science and Technology,
Nara Institute of Science and Technology, Japan
無限の可能性、ここが最先端 －Outgrow your limits－Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

2Topics
Recent advances in speech processing– ASR and TTS research– Machine Speech Chain unifies ASR and TTS– Application to code switching speechSpeech Translation– Recent Progress on Automatic Speech Interpretation

無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

3Motivation BackgroundIn human communication→ A closed-loop speech chain mechanism has a critical auditory feedback
mechanism (“Speech Chain”, Denes, Pinson 1973)

Sensory
nervesAuditory feedback

Speaking

Listening

Sensory
nervesMotornerves

無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

4Machine Speech Chain
 Proposed Method Develop a closed-loop speech chain model based on deep learning“Goodafternoon”

Use the closed-loop
for ASR and TTS
Sensory
nervesAuditory feedback“How are you?”

Speaking
Motornerves

Auditory feedback
Speaking

無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

5Machine Speech Chain

𝐿 𝑇𝑇𝑆 (𝑥, 𝑥)
ො𝐿𝐴𝑆𝑅 (𝑦, 𝑦)ොDefinition:– 𝑥 = original speech, 𝑦 = original text
– 𝑥ො = predicted speech, 𝑦ො = predicted text
– 𝐴𝑆𝑅(𝑥): 𝑥 → 𝑦ො (seq2seq model transforms speech to text)– 𝑇𝑇𝑆 𝑦 : 𝑦 → 𝑥ො (seq2seq model transforms text to speech)
Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Listening while Speaking: Speech Chain by Deep Learning”, Proc. IEEE ASRU 2017
無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

6Speech Chain with One-shot Speaker Adaptation
Proposed model– Train ASR and TTS models using unpaired data and small amount of paired data.
– Speaker individuality is generated by SPKREC embedding.

Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Machine Speech Chain with One-shot Speaker Adaptation”, Proc. INTERSPEECH 2018
無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

7

Code-switching Challenges For ASR• Typical case where paired speech and transcription are difficult to collect.
 Standard ASR is monolingual
JapaneseASR“こんにちは”EnglishASR“Hello”

 Challenge with Code-switching : Mixed multilingual input“これはstill waterですか？”Japanese
EnglishJapanese
無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

Output text
ASR?

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

8Code-switching Challenges• Typical case where paired speech and transcription are difficult to collect.
 Challenge with Code-switching : Mixed multilingual input“これはstill waterですか？”Japanese
EnglishJapanese

Output text
ASR?

Given only speech

Given only text
𝑡෧
𝑒𝑥𝑡

ASR
ASR

෧ unfold𝑡෧
𝑒𝑥𝑡
t𝑒𝑥𝑡

TTS

෧

෧

TTS

෧
𝑡𝑒𝑥𝑡

𝑠𝑝𝑒𝑒𝑐ℎASR

𝐿𝑜𝑠𝑠

𝐿𝑜𝑠𝑠𝑡𝑒𝑥𝑡

෧෧𝑡𝑒𝑥𝑡18.11% CER-> 5.08% CER

TTS

S. Nakayama, A. Tjandra, S. Sakti, S. Nakamura, “Speech chain for semi-supervised learning of Japanese-English code-switching ASR and TTS”. In Proc. IEEE SLT, 2018
無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

9Topics
Recent advances in speech processing– ASR and TTS research– Machine Speech Chain unifies ASR and TTS– Application to code switching speechSpeech Translation– Recent Progress on Automatic Speech Translation

無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Speech Translation

MultilingualSpeechRecognition

Japanese「私は学校に行く: Watashi wa Gakko he iku」
Watashi waGakko he iku

SpokenLanguage
Translation

MultilingualSpeechSynthesis

EnglishI go to school

I go to school• Cascaded process of speech recognition, machine translation, and speech synthesis.
• Machine translation of ASR transcripts.

無限の可能性、ここが最先端－Outgrow your limits－
2019/06/15

10NAIST
CLI9 Keynote Satoshi Nakamura,

http://www.naist.jp/

11

Cross-lingual Communication
Source Language

Target Language

Discource
Context

Domainknowledge

Speech“to o kyo e i ku”

MT results/I/go/to/Tokyo/

Speech⇒Text
ASR

Input:
Text
SpeechVideoGesture

Linguistic
Information

Image⇒text
PR
Text

Paralinguistic
Personality, Prosody
Emotion,Style,Personality,Prosody,Gesture

Realtime
Incremental

MT
Conversion
DialogControl

End-to-end Process

TTS results“ai go tu tokyo/

Text⇒Speech
TTS
Text⇒Image
Image Generation

Linguistic
Information

Text

Personality, Prosody

Output:Text
SpeechVideo
GestureParalinguistic
Emotion,Style,Personality,Prosody,
GestureCommunication① Simultaneity, Incremental, Latency,② +Para/non linguistic information

無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL Dec.6 2019 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

12

Human Interpreter [A.Mizuno 2016]
E-J Translation Example(1) The relief workers (2) say (3) they don’t have (4) enough food, water, shelter,and medical supplies (5) to deal with (6) the gigantic wave of refugees (7) who areransacking the countryside (8) in search of the basics (9) to stay alive.
(1) 救援担当者は (9) 生きるための (8) 食料を求めて (7) 村を荒らし回っている (6) 大量の難民達の (5)世話をするための (4) 十分な食料や水，宿泊施設，医療品が (3) 無いと (2) 言っています．

(1) 救援担当者達の (2) 話では (4)食料，水，宿泊施設，医薬品が， (3) 足りず (6) 大量の難民達の (5)世話が出来ないとのことです．(7) 難民達は今村々を荒らし回って， (9) 生きるための (8) 食料を求めているのです．

Memory Chunk

Necessary #Chunk＞３！
無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

Necessary #Chunk＜３！
LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Simultaneous Speech Translation with Adaptive Delay
Define a special token <wait><wait> <wait> President Bush <wait> meetsブッシュ 大統領はプーチンと会談

with13Putinする

Katsuki Chousa, Katsuhito Sudoh, and Satoshi Nakamura. 2019. Simultaneous Neural Machine Translation using Connectionist Temporal Classification.
arXiv preprint , 1911.1193
無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

14

Paralinguistic Speech Translation

Q. T. Do, S. Sakti, S. Nakamura, “Sequence-to-Sequence Models for Emphasis Speech Translation”. IEEE/ACM Transactions on Audio, Speech, and Language
Processing, 26(10):1873–1883, 2018
無限の可能性、ここが最先端 －Outgrow your limits－

Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

15

Summary
Machine Speech Chain– Semi-supervised training using unpaired data
– Code switching speech, under-resource language, and continuous learning

Speech Translation– Simultaneous speech translation• Segmentation, anticipation, rewording, evaluation– Paralinguistic speech translation• Emphasis, and emotions

Future works– Understanding and interpretation– Context, situation and multi-modality– Common sense, knowledge, and cross-cultural knowledge

無限の可能性、ここが最先端 －Outgrow your limits－Dec. 6. 2019

LT4ALL ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/