End-to-end approach to ASR, TTS
and Speech Translation
Satoshi Nakamura1,2
with Sakriani Sakti1,2, Andros Tjandra1,2,Takatomo Kano, and Quoc Truong Do
1Nara Institute of Science & Technology, Japan
2RIKEN, Center for Advanced Intelligence Project AIP, Japan

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

1

Outline• Machine Speech Chain• Machine Speech Chain: Listening while speaking• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Listening while Speaking: Speech Chain by Deep
Learning”, ASRU 2017

• Speech Chain with One-shot Speaker Adaptation• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura1,2 “Machine Speech Chain with One-shot
Speaker Adaptation”, Proceedings of INTERSPEECH 2018

• End-to-end Feedback Loss in Speech Chain Framework via Straight-throughEstimator• A. Tjandra, S. Sakti, S. Nakamura, "End-to-end Feedback Loss in Speech Chain Framework via
Straight-through Estimator", in Proc. ICASSP, 2019

• End-to-end Speech-to-speech Translation• Structure Based Curriculum Learning for End-to-end English-Japanese Speech Translation• Takatomo Kano, Sakriani Sakti, Satoshi Nakamura, “Structure Based Curriculum Learning for Endto-end English-Japanese Speech Translation”, INTERSPEECH2017
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

2

Outline• Machine Speech Chain• Machine Speech Chain: Listening while speaking• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Listening while Speaking: Speech Chain by Deep
Learning”, ASRU 2017

• Speech Chain with One-shot Speaker Adaptation• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura1,2 “Machine Speech Chain with One-shot
Speaker Adaptation”, Proceedings of INTERSPEECH 2018

• End-to-end Feedback Loss in Speech Chain Framework via Straight-throughEstimator• A. Tjandra, S. Sakti, S. Nakamura, "End-to-end Feedback Loss in Speech Chain Framework via
Straight-through Estimator", in Proc. ICASSP, 2019

• End-to-end Speech-to-speech Translation• Structure Based Curriculum Learning for End-to-end English-Japanese Speech Translation• Takatomo Kano, Sakriani Sakti, Satoshi Nakamura, “Structure Based Curriculum Learning for Endto-end English-Japanese Speech Translation”, INTERSPEECH2017
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

3

Motivation Background• In human communication→ A closed-loop speech chain mechanism has a critical auditory feedback mechanism→ Children who lose their hearing often have difficulty to produce clear speech

Sensory
nervesAuditory feedback

Speaking

Listening

Sensory
nervesMotornervesSatoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

4

Speech Chain: Denes, Pinson 1973

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

5Delayed Auditory Feedback*1,2• DAF:• It can consist of a device that enables a user to speak into a microphone andthen hear his or her voice in headphones a fraction of a second later• Effects in people who stutter• Those who stutter had an abnormal speech–auditory feedback loop that wascorrected or bypassed while speaking under DAF.

• Effects in normal speakers• DAF in non-stutterers to see what it can prove about the structure of the auditory
and verbal pathways in the brain.
• Indirect effects of delayed auditory feedback in non-stutterers include reduction inrate of speech, increase in intensity, and increase in fundamental frequency in orderto overcome the effects of the feedback. Direct effects include repetition of syllables,
mispronunciations, omissions, and omitted word endings.
*1Bernard S. Lee, “Delayed Speech Feedback”, The Journal of the Acoustical Society of America 22, 824 (1950);*2 Wikipedia “Delayed Auditory Feedback”
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

6

Human-Machine Interaction
 Modality in Human-Machine Interaction
 Providing a technology with ability to listen and speak“Goodafternoon”
Sensory
nervesAuditory feedback

Speaking

Speechrecognition“How are you?”Sensory
nervesListeningMotornerves“Good afternoon”
Recognized words

Speaking

SpeechSynthesis
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

7

Machine Speech Chain
 Proposed Method
 Develop a closed-loop speech chain model based on deep learning The first deep learning model that integrates human speech perception & production behaviors

Not only has the capability to listen and speak,
but also listen while speaking“Goodafternoon”
Sensory
nervesAuditory feedback“How are you?”

Speaking
MotornervesAuditory feedback
Speaking

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

8

Motivation Background• Despite the close relationship between speech perception &production → ASR and TTS researches have progressed independently
Property

ASR

TTS

Speech features

MFCC
Mel-fbank

MGC
log F0, Voice/Unvoice, BAP

Text features

PhonemeCharacterPhoneme + POS + LEX + …
(Full context label)

Model

GMM-HMM
Hybrid DNN/HMM
End-to-end ASR

GMM-HSMM
DNN-HSMM
End-to-end TTS

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

10Machine Speech Chain• Definition:• 𝑥 = original speech, 𝑦 = original text
• 𝑥ො = predicted speech, 𝑦ො = predicted text
• 𝐴𝑆𝑅(𝑥): 𝑥 → 𝑦ො (seq2seq model transforms speech to text)• 𝑇𝑇𝑆 𝑦 : 𝑦 → 𝑥ො (seq2seq model transforms text to speech)
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

11

Machine Speech Chain
Case #1: Supervised Learning with Speech-Text Data
• Given a pair speech-text 𝒙, 𝒚
• Train ASR and TTS in supervised learning• Directly optimized:→ 𝐴𝑆𝑅 by minimize ℒ𝐴𝑆𝑅 𝑦, 𝑦ො→ 𝑇𝑇𝑆 by minimizing loss between ℒ 𝑇𝑇𝑆 𝑥, 𝑥ො
• Update both ASR and TTS independently

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

12

Machine Speech Chain
Case #2: Unsupervised Learning with Text Only
• Given the unlabeled text features 𝒚
1. TTS generates speech features 𝑥ො
2. Based on 𝑥,
ො ASR tries to reconstructtext features 𝑦ො
3. Calculate ℒ𝐴𝑆𝑅 (𝑦, 𝑦)ො between originaltext features 𝑦 and the predicted 𝑦ොPossible to improve ASR with text onlyby the support of TTS

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

13

Machine Speech Chain
Case #3: Unsupervised Learning with Speech Only
• Given the unlabeled speech features 𝒙
1. ASR predicts the most possible
transcription 𝑦ො
2. Based on 𝑦,ො TTS tries to reconstructspeech features 𝑥ො
3. Calculate ℒ 𝑇𝑇𝑆 (𝑥, 𝑥)ො between original
speech features 𝑥 and the predicted 𝑥ොPossible to improve TTS with speech onlyby the support of ASR

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

14

Sequence-to-Sequence ASR
Input & output• 𝒙 = [𝑥1 , … , 𝑥𝑆 ] (speech feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑡𝑑 = decoder state at time 𝑡• 𝑎𝑡 = attention probability at time t• 𝑎𝑡 𝑠 = 𝐴𝑙𝑖𝑔𝑛 ℎ𝑠𝑒 , ℎ𝑡𝑑••

𝑎𝑡 𝑠 =

exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠𝑒 ,ℎ𝑡𝑑𝑒 𝑑σ𝑆
𝑠=1 exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠 ,ℎ𝑡
𝑐𝑡 = σ𝑆𝑠=1 𝑎𝑡 𝑠 ∗ ℎ𝑠𝑒 (expected context)

Loss function
𝑇ℒ𝐴𝑆𝑅 𝑦, 𝑝𝑦

1= − ෍ ෍ 1(𝑦𝑡 = 𝑐) ∗ log 𝑝𝑦𝑡 [𝑐]𝑇
𝑡=1 𝑐∈[1..𝐶]

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

15

Sequence-to-Sequence TTS
End of speechprediction

Fully connected

Input & output• 𝒙𝑹 = [𝑥1 , … , 𝑥𝑆 ] (linear spectrogram feature)• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] (mel spectrogram feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)
Loss function

Fully connected

CBHG: Convolution Bank + Highway + bi-GRU

1 𝑆ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො = ෍
𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2
𝑆
𝑠=1
𝑆
1
෠
ℒ 𝑇𝑇𝑆2 𝑏, 𝑏 = − ෍𝑏𝑠 log(𝑏෠𝑠 ) + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠
𝑆𝑠=1ℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

16

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

17

Experimental Set-up• Features• Speech:• 80 Mel-spectrogram (used by ASR & TTS)
• 1024-dim linear magnitude spectrogram (SFFT) (used by TTS)
• TTS reconstruct speech waveform by using Griffin-Lim
to predict the phase & inverse STFT
• Text:• Character-based prediction• a-z (26 alphabet)• 6 punctuation mark (,:’?.-)• 3 special tags 
 
 <spc> (start, end, space)

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

18

Experiments on Single-speaker• Dataset:• BTEC corpus (text), speech generated by Google TTS (using gTTS library)• Supervised training: 10000 utts (text & speech paired)• Unsupervised training: 40000 utts (text & speech unpaired)• Result:
Data

Hyperparameter
gen.
𝛼
𝛽
mode

ASR
CER
(%)

Paired
(10k)

-

-

-

+Unpaired
(40k)

0.25
0.5
0.25
0.5

1
1
1
1

greedygreedybeam 5beam 5

TTS
Mel

Raw

Acc
(%)

10.06

7.07

9.38

97.7

5.83
5.75
5.44
5.77

6.21
6.25
6.24
6.20

8.49
8.42
8.44
8.44

98.4
98.4
98.3
98.3

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

Acc: End of speechprediction accuracy

19

Experiments on Multi-speakers• Dataset• BTEC ATR-EDB corpus (text & speech) (25 male, 25 female)• Supervised training: 80 utts / spk (text & speech paired)• Unsupervised training: 360 utts / spk (text & speech unpaired)• Result

Hyperparameter
gen.
𝛼
𝛽
mode

ASR
CER
(%)

Paired(80 utt/spk)

-

-

-

+Unpaired(remaining)

0.25
0.5
0.25
0.5

1
1
1
1

greedygreedybeam 5beam 5

Data

TTS
Mel

Raw

Acc
(%)

26.47

10.21

13.18

98.6

23.03
20.91
22.55
19.99

9.14
9.31
9.36
9.20

12.86
12.88
12.77
12.84

98.7
98.6
98.6
98.6

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

Acc: End of speechprediction accuracy

20

Outline• Machine Speech Chain• Machine Speech Chain: Listening while speaking• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Listening while Speaking: Speech Chain by Deep
Learning”, ASRU 2017

• Speech Chain with One-shot Speaker Adaptation• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura1,2 “Machine Speech Chain with One-shot
Speaker Adaptation”, Proceedings of INTERSPEECH 2018

• End-to-end Feedback Loss in Speech Chain Framework via Straight-throughEstimator• A. Tjandra, S. Sakti, S. Nakamura, "End-to-end Feedback Loss in Speech Chain Framework via
Straight-through Estimator", in Proc. ICASSP, 2019

• End-to-end Speech-to-speech Translation• Structure Based Curriculum Learning for End-to-end English-Japanese Speech Translation• Takatomo Kano, Sakriani Sakti, Satoshi Nakamura, “Structure Based Curriculum Learning for Endto-end English-Japanese Speech Translation”, INTERSPEECH2017
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

21

Speech Chain with One-shot Speaker Adaptation

Andros Tjandra1,2, Sakriani Sakti1,2, Satoshi Nakamura1,2“Machine Speech Chain with One-shot Speaker
Adaptation”, Proceedings of INTERSPEECH 2018
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

16

Sequel: Speech Chain with
One-shot Speaker Adaptation• Motivation• Previous model able to improve single-speaker result significantly• Limitation: couldn’t train on unseen speaker (discrete speaker embedding)• Proposed model

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

23

One-shot Speaker Adaptation
on TTS• Instead of using discrete speaker index(one vector for one speaker)• We generate a vector given a shortutterance
by using DeepSpeaker(speaker recognition model)• Take the last layer before softmax
as embedding 𝑧• Integrate the information with Tacotron’s
decoder for generation
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

24

ASR Results
Model
CER (%)
Supervised training:
WSJ train_si84 (16hrs speech, paired) -> Baseline
Att Enc-Dec
17.35
Supervised training:
WSJ train_si284 (66 hrs speech, paired) -> UpperboundAtt Enc-Dec
7.12
Semi-supervised training:
WSJ train_si84 (paired) + train_si200 (unpaired)Label propagation (greedy)
17.52
Label propagation (beam=5)
Proposed speech chain

14.58
9.86

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

25

TTS Results Text: “the busses aren’t the problem, they actually provide a solution”• Single Speaker (LJSpeech) (p = paired, u = unpaired)
Baseline (P 30%)

Sp-Chain (P 30% + U 70%)

Full (P 100%)• Multispeaker (WSJ)
SpeakerBaseline (P SI84)

Sp-Chain (P si84 + U si200)

Full (P si284)

FemaleA

Male
B

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

26

Outline• Machine Speech Chain• Machine Speech Chain: Listening while speaking• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Listening while Speaking: Speech Chain by Deep
Learning”, ASRU 2017

• Speech Chain with One-shot Speaker Adaptation• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura1,2 “Machine Speech Chain with One-shot
Speaker Adaptation”, Proceedings of INTERSPEECH 2018

• End-to-end Feedback Loss in Speech Chain Framework via Straight-throughEstimator• A. Tjandra, S. Sakti, S. Nakamura, "End-to-end Feedback Loss in Speech Chain Framework via
Straight-through Estimator", in Proc. ICASSP, 2019

• End-to-end Speech-to-speech Translation• Structure Based Curriculum Learning for End-to-end English-Japanese Speech Translation• Takatomo Kano, Sakriani Sakti, Satoshi Nakamura, “Structure Based Curriculum Learning for Endto-end English-Japanese Speech Translation”, INTERSPEECH2017
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

27

End-to-end Feedback Loss in Speech Chain Framework
via Straight-through Estimator

Andros Tjandra1,2, Sakriani Sakti1,2, Satoshi Nakamura1,2"End-to-end Feedback Loss in Speech Chain Framework
via Straight-through Estimator", in Proc. ICASSP, 2019
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

16

Straight-Through Estimator for Speech Chain
 Proposed Approach: Handle backpropagation through discrete nodes

Feedback loss: ℒ 𝑇𝑇𝑆 (𝑥, 𝑥)
ො where 𝑥 = 𝑇𝑇𝑆(𝑦,ො 𝑧)a) Speech chain loop with speaker embedding module 𝑧b) Original: feedback ℒ 𝑇𝑇𝑆 can’t be backpropagated through variable 𝑦ොc) Proposal: Estimate gradient through variable 𝑦ො with straight-through estimator
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

29

Straight-Through Estimatora)ST-argmax
Deterministic choosing token by highest probability
𝑝𝑦𝑡 c =

exp ℎ𝑡𝑑 [𝑐]σ𝐶𝑖=1 exp ℎ𝑡𝑑 [𝑐]𝑦෤𝑡 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑐 𝑝𝑦𝑡 [𝑐]b)ST-Gumbel softmax
Sampling a token from 𝑝𝑦𝑡 𝑐 :𝑝𝑦𝑡 𝑐 =

exp (ℎ𝑡𝑑 𝑐 + 𝑔𝑐 )/𝜏𝜏 = temperature

σ𝐶𝑖=1 exp (ℎ𝑡𝑑 𝑐 + 𝑔𝑐 )/𝜏
𝑦෤𝑡 ~𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑖𝑐𝑎𝑙 𝑝𝑦𝑡 1 , . . , 𝑝𝑦𝑡 𝐶

New gradient ℒ 𝑇𝑇𝑆 w.r.t. 𝜃𝐴𝑆𝑅

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

31

Experiments on Multi-Speakers WSJ Task
 Data set• Training set: Supervised (paired text & speech)
WSJ SI-284 dataset (upperbound)(37318 utterances, ~81 h, 284 speakers)• Development set: dev93

Model• Evaluation set: eval92

CER (%)
Baseline

Enc-Dec Att-MLP [Kim et al., 2017]

11.08

Enc-Dec Att-MLP-Loc [Kim et al., 2017]

8.17

Enc-Dec Att-MLP [Tjandra et al., 2017]

7.12

Enc-Dec Att-MLP-MA (ours) [Tjandra et al., 2018]

6.43

Proposed Method
Enc-Dec Att-MLP-MA SP-Chain ST argmax

5.75

Enc-Dec Att-MLP-MA SP-Chain ST gumbel

5.70

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

32

Outline• Machine Speech Chain• Machine Speech Chain: Listening while speaking• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Listening while Speaking: Speech Chain by Deep
Learning”, ASRU 2017

• Speech Chain with One-shot Speaker Adaptation• Andros Tjandra, Sakriani Sakti, Satoshi Nakamura1,2 “Machine Speech Chain with One-shot
Speaker Adaptation”, Proceedings of INTERSPEECH 2018

• End-to-end Feedback Loss in Speech Chain Framework via Straight-throughEstimator• A. Tjandra, S. Sakti, S. Nakamura, "End-to-end Feedback Loss in Speech Chain Framework via
Straight-through Estimator", in Proc. ICASSP, 2019

• End-to-end Speech-to-speech Translation• Structure Based Curriculum Learning for End-to-end English-Japanese Speech Translation• Takatomo Kano, Sakriani Sakti, Satoshi Nakamura, “Structure Based Curriculum Learning for Endto-end English-Japanese Speech Translation”, INTERSPEECH2017
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

33

Structure Based Curriculum Learning for
End-to-end Direct English-Japanese Speech Translation

Takatomo Kano, Sakriani Sakti, Satoshi Nakamura, “Structure Based Curriculum
Learning for End-to-end English-Japanese Speech Translation”, INTERSPEECH2017
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

34/19

Traditional Speech Translation
ASR
Englishspeechi am very nervous

MT

TTS
Japanesespeech

私 は とても 緊張 して います

Traditional approach in speech-to-speech translation systems
 construct- automatic speech recognition (ASR)- machine translation (MT)- text to speech synthesis (TTS) all of which are independently trained and tuned
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

35/19

Related Works• L.Duong et al. NAACL 2016 [1]• Title: An Attentional Model for Speech Translation Without
Transcription• Spanish to English speech-to-text direct translation withattentional encoder decoder networks• Alexandre Berard et al. NIPS workshop 2016 [2]• Title: Listen and Translate: A Proof of Concept for End-to-End
Speech-to-Text Translation• French to English speech-to-text direct translation withattentional encoder decoder networks
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

36/19

Related Works[2]• End-to-end Speech-to-text translation with attentional model
Acoustic
featureTargetWord

Attention

Bi-directional
LSTM Encoder

LSTM decoder

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

37/19

Problems• Their works are only applicable for similar syntax and word order(SVO-SVO) [1,2]• For such languages, only local movements are sufficient for translation.

Spanish to English translation
attention matrix [1]French to English translation
attention matrix [2]Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

38/19

Problems• Syntactically distant language pairs (SVO versus SOV)
suffers from long-distance reordering phenomena.
朝食

は

いくら で

す

か

how

0.003 9E-04

much

2E-04 0.819 0.828 0.033 0.833 #####

is0.003 0.017 0.037 0.005 0.166 #####

the0.01 0.026 0.038 0.024 2E-04 2E-04

breakfast

0.738 0.003 7E-04 ##### ##### 0.004?

0.08 0.122 0.001 0.882 ##### 0.935

0.09 0.057 0.001 #####English to Japanese translation attention matrix

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

39/19

Proposed method• A first attempt to build direct speech-to-text direct translation system(ST) on syntactically distant language pairs• To guide the encoder-decoder attentional model to learn this difficult
problem, we proposed a structured-based curriculum learning strategy.

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

40/19

Attention-based ST with Curriculum
Learning
Phase 1Bi-LSTM
Encoder

ASR

MT

Attention

Attention
LSTM
Decoder

Bi-LSTM
Encoder

LSTM
DecoderTrain the attentional-based encoder-decoder neural network
for a standard ASR and MT task
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

41/19

Attention-based ST with Curriculum
Learning
Slow track

Fast track

Phase 2Phase 2ASR

ASR + MT

Attention

Attention

Bi-LSTM
EncoderTuning with Mean
Squared Error

LSTM
Transcoder

Bi-LSTM
Encoder

LSTM
DecoderMT

The model’s objective now is to predict
the word representation(like the MT encoder’s output)

MT
The model now predicts thecorresponding word sequence in thetarget language given the input speech

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

42/19

Attention-based ST with Curriculum
Learning
Slow track
Phase 3

ASR + MT

Attention
Bi-LSTM
Encoder

LSTM
Transcoder

LSTM
Decoder

We combine the MT attention and decoder modules to perform the speechtranslation task from the source speech sequence to the target word sequence

Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

43/19

Attention-based ST with Curriculum
Learning
Phase 1

Phase 2Phase 3ASR

ASR

ASR + MT

Slow track
Attention
Bi-LSTM
Encoder

Bi-LSTM
Encoder

LSTM
Decoder

Attention
LSTM
Transcoder

Bi-LSTM
Encoder

LSTM
Transcoder

LSTM
Decoder

ASR + MT

MT
Fast
track

Attention
Bi-LSTM
Encoder

Attention

LSTM
Decoder

Bi-LSTM
Encoder

Attention

LSTM
Encoder

Attentional-based neural trained for ASR and text-based MT tasks
and gradually train the network for end-to-end ST tasks.
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

44/19

Experimental Set-up
System settings

Data settings

ASR
Input units
Hidden units
Output units
LSTM layer depth

23
512
27293
2
MT

Source Vocabulary
Target Vocabulary & Output size

27293
33155

Input units & Embed size
Hidden units
LSTM layer Depth
Optimizer
Adam

12823
512
2

BTEC Para-text
Train utterance
45,000
Test utterance
500
BTEC Speech
Train utterance
45,000
Test utterance
500
Speech feature
F-bank 23dim
Other
We use Google TTS system to generate BTEC
speechSatoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

45/19

Translation Accuracy

 Best performance was achieved by proposed Slow Track model Surpassed the text-based MT and cascade ASR+MT systems.
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

46/19

Overall Summary:• Machine Speech Chain by ASR-TTS coupling• Machine Speech Chain: Listening while speaking• Speech Chain with One-shot Speaker Adaptation• End-to-end Feedback Loss in Speech Chain Framework via Straight-throughEstimator• End-to-end Speech-to-speech Translation• Structure Based Curriculum Learning for End-to-end English-Japanese Speech
Translation• Future Works• Multi-modal speech chain• Advanced ASR-TTS modules• Advanced MT modules• Learn human perception and cognitive process
Satoshi Nakamura @ AHCLab, NAIST, Japan | Aug. 14th 2019 CCF MTG Xining China

47