Unifying Speech Recognition
and Generation
with Machine Speech Chain
Andros Tjandra, Sakriani Sakti, Satoshi Nakamura
Nara Institute of Science & Technology, Nara, JapanRIKEN AIP, Japan
26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan1

Outline• Motivation• Machine Speech Chain• Sequence-to-Sequence ASR• Sequence-to-Sequence TTS• Experimental Setup & Results• Conclusion

26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan2

Motivation• ASR and TTS researches have progressedindependently without exerting much mutual
influence on each other.
Property

ASRTTSSpeech featuresMFCC
Mel-fbank

MGC
log F0, Voice/Unvoice, BAP

Text features

PhonemeCharacterPhoneme + POS + LEX(full context label)

GMM-HMM
Hybrid DNN/HMM
End-to-end ASR

GMM-HSMM
DNN-HSMM
End-to-end TTS

Model

26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan3

Motivation (2)• In human communication, a closed-loop speechchain mechanism has a critical auditory feedback
mechanism.

• Children who lose their hearing often havedifficulty to produce clear speech.
26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan

4

This paper proposed …
• Develop a closed-loop speech chain model basedon deep learning model• The benefit of closed-loop architecture :• Train both ASR & TTS model together• Allow us to concatenate both labeled and unlabeledspeech & text (semi-supervised learning)• In the inference stage, we could use both ASR & TTS
module independently

26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan5

Machine Speech Chain• Definition:• 𝑥 = original speech, 𝑦 = original text
• 𝑥ො = predicted speech, 𝑦ො = predicted text
• 𝐴𝑆𝑅(𝑥): 𝑥 → 𝑦ො (seq2seq model transform speech to text)• 𝑇𝑇𝑆 𝑦 : 𝑦 → 𝑥ො (seq2seq model transform text to speech)
26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan6

Machine Speech Chain (2)• Case #1: Supervised training• We have a pair speech-text 𝑥, 𝑦
• Therefore we could directly optimized 𝐴𝑆𝑅 by minimize
𝐿𝑜𝑠𝑠𝐴𝑆𝑅 𝑦, 𝑦ො• and 𝑇𝑇𝑆 by minimizing loss between 𝐿𝑜𝑠𝑠𝑇𝑇𝑆 𝑥, 𝑥ො
𝑦 = “texts”

𝑥=𝐿𝑜𝑠𝑠𝐴𝑆𝑅 𝑦, 𝑦ො

𝐿𝑜𝑠𝑠𝑇𝑇𝑆 𝑥, 𝑥ො𝑦ො = “tex”

𝑥ො =ASR𝑥=
26-Mar-19

TTS

𝑦 = “text”
Andros Tjandra @ AHCLab, NAIST, Japan7

Machine Speech Chain (2)• Case #2: Unsupervised training
with speech only
1. Given the unlabeled speechfeatures 𝑥
2. ASR predicts most possible
transcription 𝑦ො
3. TTS based on 𝑦ො tries to reconstructspeech features 𝑥ො
4. Calculate 𝐿𝑜𝑠𝑠𝑇𝑇𝑆 (𝑥, 𝑥)ො between
original speech features 𝑥 andpredicted 𝑥ො
26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan𝑥ො =𝐿𝑜𝑠𝑠𝑇𝑇𝑆(𝑥, 𝑥)ොTTS𝑦ො = “text”
ASR𝑥=8

Machine Speech Chain (2)• Case #3: Unsupervised training withtext only
1. Given the unlabeled text features 𝑦
2. TTS generates speech features 𝑥ො
3. ASR given 𝑥ො tries to reconstructspeech features 𝑦ො
4. Calculate 𝐿𝑜𝑠𝑠𝐴𝑆𝑅 (𝑦, 𝑦)ො between
original text 𝑦 and predicted 𝑦ො𝑦ො = “text”

𝐿𝑜𝑠𝑠𝐴𝑆𝑅(𝑦, 𝑝𝑦 )ASR𝑥ො =TTS𝑦 = “text”

26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan

9

Sequence-to-Sequence ASR
Input & output• 𝒙 = [𝑥1 , … , 𝑥𝑆 ] (speech feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑡𝑑 = decoder state at time 𝑡• 𝑎𝑡 = attention probability at time t• 𝑎𝑡 𝑠 = 𝐴𝑙𝑖𝑔𝑛 ℎ𝑠𝑒 , ℎ𝑡𝑑••𝑎𝑡 𝑠 =exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠𝑒 ,ℎ𝑡𝑑𝑒 𝑑σ𝑆
𝑠=1 exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠 ,ℎ𝑡
𝑐𝑡 = σ𝑆𝑠=1 𝑎𝑡 𝑠 ∗ ℎ𝑠𝑒 (expected context)

Loss function
𝑇ℒ𝐴𝑆𝑅 𝑦, 𝑝𝑦

1= − ෍ ෍ 1(𝑦𝑡 = 𝑐) ∗ log 𝑝𝑦𝑡 [𝑐]𝑇
𝑡=1 𝑐∈[1..𝐶]

26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan10

Sequence-to-Sequence TTS
Input & output• 𝒙𝑹 = [𝑥1 , … , 𝑥𝑆 ] (linear spectrogram feature)• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] (mel spectrogram feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)
Loss function
1 𝑆ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො = ෍𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2𝑆
𝑠=1𝑆
1ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠ = − ෍𝑏𝑠 log(𝑏෠𝑠 ) + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠
𝑆𝑠=1ℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠
26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan

11Settings• Features• Speech:• 80 Mel-spectrogram (used by ASR & TTS)
• 1024-dim linear magnitude spectrogram (SFFT) (used by TTS)
• TTS reconstruct speech waveform by using Griffin-Lim topredict the phase & inverse STFT

• Text:• Character-based prediction• a-z (26 alphabet)• 6 punctuation mark (,:’?.-)• 3 special tags 
 
 <spc> (start, end, space)

26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan

12

Experiment on Single-Speaker• Dataset• BTEC corpus (text), speech generated by Google TTS (usinggTTS library)• Supervised training: 10000 utts (text & speech paired)• Unsupervised training: 40000 utts (text & speech unpaired)• ResultDataASRCER(%)

TTS

𝛼𝛽gen.
mode---

0.25

1

greedy5.83

6.21 8.49 98.4

+Unpaired 0.5
(40k)
0.25

1

greedy5.75

6.25 8.42 98.4

1

beam 5

5.44

6.24 8.44 98.3

0.5

1

beam 5

5.77

6.20 8.44 98.3

Paired(10k)26-Mar-19

Hyperparameter

MelRaw

Acc(%)

10.06 7.07 9.38 97.7

Andros Tjandra @ AHCLab, NAIST, Japan13

Experiment on Multi-Speaker Task
• Dataset• BTEC ATR-EDB corpus (text & speech) (25 male, 25 female)• Supervised training: 80 utts / spk (text & speech paired)• Unsupervised training: 360 utts / spk (text & speechunpaired)• Result
DataPaired(80 utt/spk)

Hyperparameter

ASR
CER
(%)

𝛼𝛽gen.
mode---

0.25 1Mel

Raw

Acc
(%)

26.47 10.21 13.18 98.6

greedy 23.03

9.14

12.86 98.7

+Unpaired 0.5 1 greedy 20.91(remaining) 0.25 1 beam 5 22.55

9.31

12.88 98.6

9.36

12.77 98.6

9.20

12.84 98.60.526-Mar-19

TTS

1 beam 5 19.99

Andros Tjandra @ AHCLab, NAIST, Japan

14

Conclusion• Proposed a speech chain based on deep-learningmodel• Explored applications in single and multi-speakertasks• Results: improved ASR & TTS performance byteaching each other using only unpaired data
• Future work: Perform real-time feedback
mechanisms similar to human approach

26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan15

 Thank you for listening 

26-Mar-19

Andros Tjandra @ AHCLab, NAIST, Japan19