1Recent Advances in Speech Processingand Machine Translation Research at NAISTDr. Satoshi Nakamura
Director, Data Science Center,Professor, Graduate School of Science and Technology,Nara Institute of Science and Technology
Team Leader, Tourism Information Analytics Team, AIP Center, Riken, Japan

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

3

NAIST 2011-

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

4

Where is NAIST?
500 km from Tokyo

30 mins from Osaka
90 mins fromKansai Airport

30 mins from Kyoto
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

5Campus
Land area:
139,967 m2
Constructed area:
27,392 m2
Extended area:
99,109 m2
２００ｍ

Division ofInformation Science(IS)Division of
Biological Sciences(BS)

無限の可能性、ここが最先端 －Outgrow your limits－

Division ofMaterials Science(MS)Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

6

ChronologyOct1991Apr Apr
1993 1994

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

Apr1998

Oct2011

Apr. Apr.
2017 2018©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

7

New Data Science Program for Master Students

Data science will be the BASIS of all programs
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

8

The Organization of DSC
Data Science

Director
S.Nakamura

Director ofResearchMaterials
Informatics

K.Funatsu(NAIST & U.Tokyo)

Bioinformatics

S.Nakamura, Augmented Human Communication
S.Kanaya, Systems Biology
K.Ikeda, Mathematical Informatics
Y.Matsumoto, Computational Linguistics
N.Ono, Data Science, Systems Biology
K.Sudo, Augmented Human Communication
Y.Suzuki, Data Analytics, Data Engineering
K.Yasuda, Natural Language Processing
H.Tanaka, Augmented Human Communication
R. Eguchi, Material Informatics
K.Funatsu, Chemo-infomatics
Y.Uraoka, Information Device Science
Y.Ishikawa, Information Device Science
M.Hatanaka, Materials Informatics
T. Miyao, Chemo-informatics
H.Mori (Systems Microbiology)
A.Muto (Systems Microbiology)
Y.Sakumura (Computational Biology)
K.Kunida (Computational Biology)

Open Innovation

InternationalCollaboration
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

H.Nojima (Strategy and Planning)

Underline: main affiliation©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

9Research Topics at AHC-lab

Speech Translation
Neural MachineTranslation
Multi-language ASR, TTS
Machine Speech Chain

Simultaneous Speech Translation
Project 2017-2021

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

10Research Topics at AHC-lab

Speech Translation
Neural MachineTranslation
Multi-language ASR, TTS
Machine Speech Chain

I’m looking
for a lab.

Why don’t you
join our lab!

Spoken Dialog
Multi-modal Dialog
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

Natural Language
Processing
Goal-oriented Dialog
Non goal-oriented Dialog
Deep NeuralNetwork©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

11Research Topics at AHC-lab
Brain Analysis
Incongruity measurement
Cognitive Load
EEG Hyper Scanning
Speech Translation
Neural Machine
ANR-CREST:TAPASTranslationSST&CBT
Multi-language ECA
ASR, TTS
MachineSpeech Chain
2019-2024

Affective Computing
SST, CBT,I’m looking
Early Dementia
for a lab.

Why don’t you
join our lab!

Spoken Dialog
Multi-modal Dialog
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

Natural Language
Processing
Goal-oriented Dialog
Non goal-oriented Dialog
Deep NeuralNetwork©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

12

Research Topics at AHC-lab
BrainAnalysis
Incongruity measurement
RIKEN
AIP
Cognitive Load
EEG Hyper Scanning
Tourism Information
Analytics PJSpeech Translation
Neural MachineTranslation
Multi-language ASR, TTS
Machine Speech Chain

Affective Computing
SST, CBT,I’m looking
Early Dementia
for a lab.

2017-2026
Data Analytics
CaptionGeneration
Why don’t you
join our lab!

Spoken Dialog
Multi-modal Dialog
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

Natural Language
Processing
Goal-oriented Dialog
Non goal-oriented Dialog
Data Science
Center （2017-）
Deep NeuralNetwork©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

13Topics
Recent advances in speech processing– ASR and TTS research– Machine Speech Chain unifies ASR and TTS– Single speaker to multi-speaker

Speech Translation– Direct speech to text translation by DNN

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

14

Recent Progress of ASR
Traditional Technologies– Template Matching, Dynamic Programing [Sakoe 71]– Hidden Markov Modeling, N-Gram Model [Mercer 83, etc]
– Neural Network、TDNN[Waibel 89], LSTM [Hochreiter 97]– Weighted Finite State Transducer [Mohri 2006]– Big Training Data, Data Collection through Trial Service

Deep Learning (Hinton visited MSR)
– DNN-HMM [Hinton 2012]• Estimate State Posterior Probability by DNN– Connectionist Temporal Classification [Graves 2013]• Predict Phoneme Label every frame– Listen, Attend, and Spell [Chan 2016]• CTC+Attention: End-to-end modeling
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

15

Recent Performance
Saon, et al. “English Conversational Telephone Speech Recognition
by Humans and Machines”, INTERSPEECH 2017

[1] R. P. Lippmann, “Speech recognition by machines and humans,”
Speech communication, vol. 22, no. 1, pp. 1–15, 1997.

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

16

Recent Speech Synthesis
Formant-based Synthesis, Waveform Concatenation
Statistical Speech Synthesis: HTS– Speech Synthesis by HMM• Tokuda, et al., “Speech parameter generation algorithms for HMM-based speech synthesis”, ICASSP 2000

WaveNet– Waveform Convolution• van den Oord et al., “WAVENET: A GENERATIVE MODEL FOR RAW AUDIO”,
arXiv:1609.03499v2 [cs.SD] 19 Sep 2016

Tacotron– End-to-end speech synthesis with character input. Waveform generation by Griffin-Lim• Wang, et al., “TACOTRON: TOWARDS END-TO-END SPEECH SYNTHESIS”,
arXiv:1703.10135v2 [cs.CL] 6 Apr 2017

Tacotron2:– Tacotron + WaveNet
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

17Outline
Machine Speech Chain– Machine Speech Chain: Listening while speaking• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Listening while Speaking:Speech Chain by Deep Learning”, ASRU 2017
– Speech Chain with One-shot Speaker Adaptation• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura1,2 “Machine Speech Chainwith One-shot Speaker Adaptation”, Proceedings of INTERSPEECH 2018

End-to-end Speech Translation– Structure Based Curriculum Learning for End-to-end English-Japanese
Speech Translation• Takatomo Kano, Sakriani Sakti, Satoshi Nakamura, “Structure Based Curriculum
Learning for End-to-end English-Japanese Speech Translation”,
INTERSPEECH2017

無限の可能性、ここが最先端 －Outgrow your limits－©Satoshi Nakamura,Nov. 14 AHC
2019Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Motivation Background18

In human communication→ A closed-loop speech chain mechanism has a critical auditory feedback
mechanism→ Children who lose their hearing often have difficulty to produce clear speech
Sensory
nervesAuditory feedback

Speaking

Listening

Sensory
nervesMotornerves

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

19Speech Chain: Denes, Pinson 1973

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

20

Delayed Auditory Feedback

*1,2

DAF:
– Device that enables a user to speak into a microphone and then hearin headphones a fraction of a second later
Effects by DAF to people who stutter
Effects in normal speakers– DAF effects prove about the structure of the auditory and verbal pathways in thebrain.
– Indirect effects include reduction inrate of speech, increase in intensity, and increase in fundamental frequency– Direct effects include repetition of syllables, mispronunciations, omissions, andomitted word endings.

*1Bernard S. Lee, “Delayed Speech Feedback”, The Journal of the Acoustical Society of America 22, 824 (1950);*2 Wikipedia “Delayed Auditory Feedback”
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

21Machine Speech Chain
 Proposed Method

 Develop a closed-loop speech chain model based on deep learningNot only has the capability to listen and speak,
but also listen while speaking“Goodafternoon”
Sensory
nervesAuditory feedback“How are you?”

Speaking
Motornerves

Auditory feedback
Speaking

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Machine Speech Chain

22

Definition:– 𝑥 = original speech, 𝑦 = original text
– 𝑥ො = predicted speech, 𝑦ො = predicted text
– 𝐴𝑆𝑅(𝑥): 𝑥 → 𝑦ො (seq2seq model transforms speech to text)– 𝑇𝑇𝑆 𝑦 : 𝑦 → 𝑥ො (seq2seq model transforms text to speech)

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Machine Speech Chain

23

Case #1: Supervised Learning with Speech-Text Data
Given a pair speech-text 𝒙, 𝒚
– Train ASR and TTS in supervised learning– Directly optimized:→ 𝐴𝑆𝑅 by minimize ℒ𝐴𝑆𝑅 𝑦, 𝑦ො→ 𝑇𝑇𝑆 by minimizing loss between ℒ 𝑇𝑇𝑆 𝑥, 𝑥ො
– Update both ASR and TTS independently

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan.

AIST 2019 @IGDTUW India

http://www.naist.jp/

24Machine Speech Chain
Case #2: Unsupervised Learning with Text Only
– Given the unlabeled text features 𝒚
1.
2.
3.

TTS generates speech features 𝑥ො
Based on 𝑥,
ො ASR tries to reconstructtext features 𝑦ොCalculate ℒ𝐴𝑆𝑅 (𝑦, 𝑦)ො between originaltext features 𝑦 and the predicted 𝑦ොPossible to improve ASR with text onlyby the support of TTS

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW Indiahttp://www.naist.jp/

25

Machine Speech Chain
Case #3: Unsupervised Learning with Speech Only
– Given the unlabeled speech features 𝒙
1.
2.
3.

ASR predicts the most possible
transcription 𝑦ොBased on 𝑦,ො TTS tries to reconstructspeech features 𝑥ො
Calculate ℒ 𝑇𝑇𝑆 (𝑥, 𝑥)ො between original
speech features 𝑥 and the predicted 𝑥ොPossible to improve TTS with speech onlyby the support of ASR

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Sequence-to-Sequence ASR

26

Input & output• 𝒙 = [𝑥1 , … , 𝑥𝑆 ] (speech feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑡𝑑 = decoder state at time 𝑡• 𝑎𝑡 = attention probability at time t• 𝑎𝑡 𝑠 = 𝐴𝑙𝑖𝑔𝑛 ℎ𝑠𝑒 , ℎ𝑡𝑑••

𝑎𝑡 𝑠 =exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠𝑒 ,ℎ𝑡𝑑𝑒 𝑑σ𝑆
𝑠=1 exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠 ,ℎ𝑡
𝑐𝑡 = σ𝑆𝑠=1 𝑎𝑡 𝑠 ∗ ℎ𝑠𝑒 (expected context)

Loss function
𝑇ℒ𝐴𝑆𝑅 𝑦, 𝑝𝑦

1= − ෍ ෍ 1(𝑦𝑡 = 𝑐) ∗ log 𝑝𝑦𝑡 [𝑐]𝑇𝑡=1 𝑐∈[1..𝐶]

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Sequence-to-Sequence TTS
End of speechprediction

Fully connected

27

Input & output• 𝒙𝑹 = [𝑥1 , … , 𝑥𝑆 ] (linear spectrogram feature)• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] (mel spectrogram feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)
Loss function

Fully connected

1 𝑆ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො = ෍𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2𝑆
𝑠=1𝑆1෠ℒ 𝑇𝑇𝑆2 𝑏, 𝑏 = − ෍𝑏𝑠 log(𝑏෠𝑠 ) + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠
𝑆𝑠=1ℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠

CBHG: Convolution Bank + Highway + bi-GRU
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan.

AIST 2019 @IGDTUW India

http://www.naist.jp/

28

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Experimental Set-up

29Features– Speech:• 80 Mel-spectrogram (used by ASR & TTS)
• 1024-dim linear magnitude spectrogram (SFFT) (used by TTS)
• TTS reconstruct speech waveform by using Griffin-Lim
to predict the phase & inverse STFT
– Text:• Character-based prediction– a-z (26 alphabet)– 6 punctuation mark (,:’?.-)– 3 special tags 
 
 <spc> (start, end, space)

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Experiments on Single-speaker

30

Dataset:– BTEC corpus (text), speech generated by Google TTS (using gTTS library)– Supervised training: 10000 utts (text & speech paired)– Unsupervised training: 40000 utts (text & speech unpaired)

Result:
DataHyperparametergen.
𝛼𝛽mode

ASR
CER
(%)

Paired(10k)---+Unpaired
(40k)

0.25
0.5
0.25
0.5

11
1
1

greedygreedybeam 5beam 5

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019TTSMel

Raw

Acc
(%)

10.06

7.07

9.38

97.7

5.83
5.75
5.44
5.77

6.21
6.25
6.24
6.20

8.49
8.42
8.44
8.44

98.4
98.4
98.3
98.3

Acc: End of speechprediction accuracy©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Experiments on Multi-speakers

31

Dataset– BTEC ATR-EDB corpus (text & speech) (25 male, 25 female)– Supervised training: 80 utts / spk (text & speech paired)– Unsupervised training: 360 utts / spk (text & speech unpaired)

ResultHyperparametergen.
𝛼𝛽mode

ASR
CER
(%)

Paired(80 utt/spk)---+Unpaired(remaining)

0.25
0.5
0.25
0.5

111
1greedy
greedybeam 5beam 5Data

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

TTSMel

Raw

Acc
(%)

26.47

10.21

13.18

98.6

23.03
20.91
22.55
19.99

9.14
9.31
9.36
9.20

12.86
12.88
12.77
12.84

98.7
98.6
98.6
98.6

Acc: End of speechprediction accuracy©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

16Speech Chain with One-shot Speaker Adaptation

Andros Tjandra1,2, Sakriani Sakti1,2, Satoshi Nakamura1,2“Machine Speech Chain with One-shot SpeakerAdaptation”, Proceedings of INTERSPEECH 2018
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

Sequel: Speech Chain withOne-shot Speaker Adaptation

33

Motivation– Previous model able to improve single-speaker result significantly– Limitation: couldn’t train on unseen speaker (discrete speaker embedding)

Proposed model

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

34

One-shot Speaker Adaptation on TTS
Instead of using discrete speaker index(one vector for one speaker)
We generate a vector given a short utterance
by using DeepSpeaker(speaker recognition model)
Take the last layer before softmax
as embedding 𝑧
Integrate the information with Tacotron’s
decoder for generation

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

ASR Results

35

ModelCER (%)
Supervised training:
WSJ train_si84 (16hrs speech, paired) -> Baseline
Att Enc-Dec
17.35
Supervised training:
WSJ train_si284 (66 hrs speech, paired) -> UpperboundAtt Enc-Dec
7.12
Semi-supervised training:
WSJ train_si84 (paired) + train_si200 (unpaired)Label propagation (greedy)
17.52
Label propagation (beam=5)
Proposed speech chain
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

14.58
9.86
©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

36Topics
Recent advances in speech processing– ASR and TTS research– Machine Speech Chain unifies ASR and TTS– Single speaker to multi-speakerSpeech Translation– Direct speech to text translation by DNN

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

37/19Structure Based Curriculum Learning for
End-to-end Direct English-Japanese SpeechTranslation
Takatomo Kano, Sakriani Sakti, Satoshi Nakamura, “Structure Based Curriculum
Learning for End-to-end English-Japanese Speech Translation”,
INTERSPEECH2017

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

38

Recent MT progress
Rule-based MT：
Linguists generate translation rules
Corpus-based MT:– Example-Based: Automatic rule extraction from corpus[M.Nagao84, Sato et.al.,89, Sumita et. al., 91 ]– Statistical MT: Statistical Modeling of MT. Extraction of model parameters from corpus and MT
based on Noisy Channel Model [P.F.Brown, et.al. 93]– Phrase-base SMT [Koen+ 2003]

Tree-to-string– Statistical MT based on Tree Structure

Neural Machine Translation– Combination of Encoder and Decoder by LSTM [Sutskever+ 14]

Attention NMT [Bahdanau+ 15]– Add Attention to encoder and decoder

Self Attention NMT [Vaswani+ 17]– Self attention by multiple heads. Transformer.
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

39/19Traditional Speech Translation

ASR
Englishspeech

MTi am very nervousTTSJapanesespeech

私 は とても 緊張 して います

Traditional approach in speech-to-speech translation systems
 construct- automatic speech recognition (ASR)- machine translation (MT)- text to speech synthesis (TTS) all of which are independently trained and tuned
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

40/19Related Works
L.Duong et al. NAACL 2016 [1]– Title: An Attentional Model for Speech Translation Without Transcription– Spanish to English speech-to-text direct translation with attentional encoder decoder networks

Alexandre Berard et al. NIPS workshop 2016 [2]– Title: Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation– French to English speech-to-text direct translation with attentional encoder decoder networks

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

41/19Related Works[2]End-to-end Speech-to-text translation with attentional model

Acoustic
featureTarget
Word

Attention

Bi-directional
LSTM Encoder

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

LSTM decoder©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

42/19Problems
Their works are only applicable for similar syntax and word order (SVO-SVO) [1,2]For such languages, only local movements are sufficient for translation.

Spanish to English translation
attention matrix [1]
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

French to English translation
attention matrix [2]©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

43/19Problems• Syntactically distant language pairs (SVO versus SOV)
suffers from long-distance reordering phenomena.
朝食はいくら ですかhow

0.003 9E-04

much

2E-04 0.819 0.828 0.033 0.833 #####

is0.003 0.017 0.037 0.005 0.166 #####

the0.01 0.026 0.038 0.024 2E-04 2E-04

breakfast

0.738 0.003 7E-04 ##### ##### 0.004?

0.08 0.122 0.001 0.882 ##### 0.935

0.09 0.057 0.001 #####English to Japanese translation attention matrix

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

44/19Proposed method• A first attempt to build direct speech-to-text direct translation system(ST) on syntactically distant language pairs• To guide the encoder-decoder attentional model to learn this difficult
problem, we proposed a structured-based curriculum learning strategy.

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

45/19Attention-based ST with Curriculum Learning

Phase 1Bi-LSTM
EncoderASRMTAttention

Attention
LSTM
Decoder

Bi-LSTM
Encoder

LSTM
DecoderTrain the attentional-based encoder-decoder neural network
for a standard ASR and MT task

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

46/19Attention-based ST with Curriculum Learning

Slow track

Fast track

Phase 2Phase 2ASR

ASR + MT

Attention

Attention

Bi-LSTM
EncoderTuning with Mean
Squared Error

LSTM
Transcoder

MT

LSTM
DecoderMT

The model’s objective now is to predict
the word representation(like the MT encoder’s output)
無限の可能性、ここが最先端 －Outgrow your limits－

Bi-LSTM
Encoder

Nov. 14 2019

The model now predicts thecorresponding word sequence in thetarget language given the input speech©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

47/19Attention-based ST with Curriculum Learning

Slow track
Phase 3

ASR + MT

Attention
Bi-LSTM
Encoder

LSTM
Transcoder

LSTM
Decoder

We combine the MT attention and decoder modules to perform the speechtranslation task from the source speech sequence to the target word sequence

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

48/19Attention-based ST with Curriculum Learning

Phase 1

Phase 2Phase 3ASR

ASR

ASR + MT

Slow track
Attention
Bi-LSTM
Encoder

Bi-LSTM
Encoder

LSTM
Decoder

Attention
LSTM
Transcoder

Bi-LSTM
Encoder

LSTM
Transcoder

LSTM
Decoder

ASR + MTMT
Fasttrack

Attention
Bi-LSTM
Encoder

Attention

LSTM
Decoder

Bi-LSTM
Encoder

Attention

LSTM
Encoder

Attentional-based neural trained for ASR and text-based MT tasks
and gradually train the network for end-to-end ST tasks.
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

49/19Experimental Set-up
System settings

Data settings

ASR
Input units
Hidden units
Output units
LSTM layer depth

23512
27293
2
MTSource Vocabulary
Target Vocabulary & Output size

27293
33155

Input units & Embed size
Hidden units
LSTM layer Depth
Optimizer
Adam12823
5122

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

BTEC Para-text
Train utterance
45,000
Test utterance
500
BTEC Speech
Train utterance
45,000
Test utterance
500
Speech featureF-bank 23dim
Other
We use Google TTS system to generate BTECspeech©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

50/19Translation Accuracy

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/

51

Overall Summary:
Recent advances in speech processing
ASR and TTS research at NAISTMachine Speech Chain unifies ASR and TTS
Single speaker to multi-speaker

Speech Translation research at NAIST
Direct speech to text translation by DNN– Structure Based Curriculum Learning for End-to-end English-Japanese Speech
Translation

Future Works– Advanced MT modules by Deep Learning– Learn human perception and cognitive process
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 14 2019

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2019 @IGDTUW India

http://www.naist.jp/