Machine Speech Chain:
A Machine that Learned to Listen, Speak, andwhile

Dr.-Ing. Sakriani Sakti
Research Associate Professor of
Nara Institute of Science and Technology (NAIST), Japan
Research Scientist of RIKEN Center for
Advanced Intelligence Project AIP (RIKEN AIP), Japan

Co-Authors: Andros Tjandra, Johanes Effendi, Sahoko Nakayama, Sashi Novitasari, Satoshi Nakamura

1

Human Communication

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 20202

Human-to-Human Communication▪ Speech in Human Communication→ The most natural modality to express & share their ideas, experiences, and knowledge
Business

Conversations

Meeting

LectureSakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 20203

How do We Communicate?
▪ Speech Chain [Denes & Pinson, 1993]“Good
afternoon”

Sensory
nervesAuditory feedback
Listening

Speaking
Motornerves

Linguistic
Level

Physiological
Level

Acoustic
Level

Physiological
Level

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

Sensory
nerves

Linguistic
Level

4

How do We Speak?
▪ Speech Production Model→ By expelling air from the lungs through the trachea, and passed throughthe larynx then out the mouth or nose (vocal tract)h(t)Source (Glotal Pulse)x(t)Vocal tract
filterSpeech Soundy(t) = x(t)*h(t)[B]
Vocal tract[C]
Speech sound[A]
Vocal folds

Air flow from lungs

[A]

[B]

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

[C]

5

Speech Utterances[Phonical, 2017]

F3
F2
F1

Formant Frequencies“She just had a baby”[Source: https://web.stanford.edu/class/cs224s/lectures/224s.17.lec2.pdf|

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 20206How do We Hear?
▪ Human Ear→ Receive the acoustic waves, amplify the intensity, & analyze the frequency
The cochlea in inner ear[Bosi & Goldberg, 2003]

directional
microphone

impedance matching,overload protection

neural encoding,frequency analysis

Separate Sound by Frequency[Source: http://hyperphysics.phy-astr.gsu.edu/hbase/Sound/cochimp.html|

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 20207

Human-Machine Interaction

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 20208

Human-Machine Interaction▪ Modality in Human-Machine Interaction

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 20209

Human-Machine Interaction▪ Modality in Human-Machine Interaction→ One of the earliest objectives in artificial intelligence (AI) has been to realize

a technology or a machine that can communicate with the human

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202010

Human-Machine Interaction▪ Modality in Human-Machine Interaction→ Providing a technology with ability to listen and speak“Goodafternoon”
Sensory
nervesAuditory feedback

Speaking

Speechrecognition“How are you?”Sensory
nervesListeningMotornerves“Good afternoon”
Recognized words

Speaking

SpeechSynthesis
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202011

Automatic Speech Recognition (ASR)▪ Traditional ASR based on Hidden Markov Model (HMM)“MY SPEECH”
FeatureExtraction/my speech/

SearchAlgorithm

Speech Signal/my//m/ /a/X1X2X3X4 …

/speech/

/i/

/s/

/p/

/ee/

The most probable string of words

PhonemeWord

Sentence

Acoustic Model

LexiconLanguage Model/t/ /sh/…XT-1XT

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202012

Text-to-Speech Synthesis (TTS)▪ Traditional TTS based on Hidden Markov Model (HMM)

X1X2X3X4…

/m/ /a/…XT-1XT

/i/

/s/

/p/

/ee/

/my/

/t/

/sh/

/speech//my speech/“MY SPEECH”[Zen et al., 2009]
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

13

ASR and TTS Performance

TTS: From robot voice tohuman-like voice[Source: https://www.economist.com/technology-quarterly/2017-05-01/language]
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

14

ASR and TTS Performance

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202015

Paradigm Shift: Deep Learning Hype

[Source: Linked IN | Machine Learning vs Deep Learning][Source: https://www.gartner.com/en/newsroom/press-releases/2017-08-15-gartneridentifies-three-megatrends-that-will-drive-digital-business-into-the-next-decade]
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202016

Recent ASR Technology▪ ASR based on Deep Learning“MY SPEECH”“MY SPEECH”/my speech//my//m/ /a/X1X2X3X4…

/speech/

/i/

/s/

/p/

/ee/

Important factors of
Deep Learning:

Deep Learning/t/→ Simplify many complicatedhand-engineered models→ Let the networks find the waythat map from speech to text/sh/

…XT-1XT

X1X2X3X4…

…XT-1XT

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202017

Recent ASR Technology▪ ASR based on Deep Learning“MY SPEECH”

Input and Output• 𝒙 = [𝑥1 , … , 𝑥𝑆 ] (Speech features)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (Text)

Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑡𝑑 = decoder state at time 𝑡• 𝑎𝑡 = attention probability

Deep Learning

X1 X2X3X4…

…XT-1XT

NN types• LSTM (Long short-term memory)• Bi-LSTM (Bidirectional LSTM)

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

18

ASR Progress[Source: https://www.economist.com/technology-quarterly/2017-05-01/language]
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

19

ASR Progress▪ IBM vs Microsoft:“Human parity”speech recognition record→ Makes the same / fewer errors than professional transcriptionists

New IBM System[Xiaong et al., 2017][Saon et al., 2017]
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202020

Recent TTS Technology▪ TTS based on Deep Learning

X 1X2X3X 4…

/m/ /a/…XT-1XT

/i/

/s/

/p/

/ee/

/my/

/speech//t/

X1X2X 3X4…

…XT-1XT/sh/Deep Learning/my speech/“MY SPEECH”

Important factors of
Deep Learning:→ Simplify many complicatedhand-engineered models→ Let the networks find the waythat map from text to speech“MY SPEECH”

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202021

Recent TTS Technology▪ TTS based on Deep Learning

X1X2X3X4…

…XT-1XT

Input and Output• 𝒙𝑹 = [𝑥1 , … , 𝑥𝑆 ] (linear spect. Feat.)• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] (mel spect. feat)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability

Deep Learning“MY SPEECH”

NN types• FC (Full-connected)• LSTM (Long short-term memory)• Bi-LSTM (Bidirectional LSTM)• CBHG (Conv bank + highway net +bidirectional GRU)
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

22

TTS Progress▪ Google's DeepMind:
Major milestone in making machines talk like humans

WaveNet: Generative Model for Raw Audio[Source: https://www.zdnet.com/article/googles-deepmind-claims-majormilestone-in-making-machines-talk-like-humans/]

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

23

TTS Progress▪ Google Duplex:
AI System for Accomplishing Real-World Tasks Over the Phone

Duplex scheduling a hair salon appointment:

Duplex calling a restaurant:[Source: https://ai.googleblog.com/2018/05/
duplex-ai-system-for-natural-conversation.html]
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202024

What is left?
Are all problems solved?

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202025

Machine Learning vs Human Learning▪ Learning Issues→ It requires a lot of parallel speech and text, more than human need
→ Such data is often not available

SpeechCorpus• Good Morning• Have a nice day.
Basic

Text
Corpus• I lost my passport! Travel• Where is the station?

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

• How much is this?
•Ten dollars.
Shopping

26

Machine Learning vs Human Learning▪ Learning Issues→ It requires a lot of parallel speech and text, more than human need
→ Such data is often not available
Area
AfricaAmericas
Asia
EuropePacific
TotalsLiving Languages
Count
Percent
2,110
30.5
993
14.4
2,322
33.6
234
3.4
1,250
18.1
6,909
100

Number of Speakers
Count
Percent
726,453,403
12.2
50,496,321
0.8
3,622,771,264
60.8
1,553,360,941
26.1
6,429,788
0.1
5,959,511,717
100

Only up to ~100 languages are covered bylanguage technologies.

Lewis, M. Paul (ed.), 2009. Ethnologue: Languages of the World, Sixteenth edition.
Dallas, Tex.: SIL International. Online version: http://www.ethnologue.com/.

Nearly 7000 living languages (spoken by 350
million people) have not yet been covered.

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202027

Machine Learning vs Human Learning▪ Human Learning→ Humans learn how to talk by constantly repeating their articulations & listening to sounds produced→ A closed-loop speech chain mechanism has a critical auditory feedback mechanism
Children who lose their hearing often
have difficulty to produce clear speech“Goodafternoon”
Sensory
nervesAuditory feedback

Adults who become deaf
after becoming proficient with a language
nonetheless suffer speech articulation declines
as a result of the lack of auditory feedback

Speaking[Waldstein, 1990]

MotornervesSakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202028

Machine Learning vs Human Learning▪ Human Brain: Sensorimotor Integration in Speech Processing(1) the auditory system is critically involved in the production of speech(2) the motor system is critically involved in the perception of speech

Spt exhibits sensorimotor response properties, activating
both during the passive perception of speech and during
covert (subvocal) speech articulation [Hickok et al, 2003]

An Integrated State Feedback Control (SFC) Modelof Speech Production [Hickok et al. 2011]

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202029

Machine Learning vs Human Learning▪ Machine Learning→ Computers are able to learn how to listen or learn how to speak→ But, computers cannot hear their own voice

Listening“Good afternoon”
Recognized words“How are you?”

Speech recognition→ Only listening
Speaking

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

Speech synthesis→ Only speaking

30

Part I
Basic Machine Speech Chain[A. Tjandra, S. Sakti, S. Nakamura, "Listening while Speaking:Speech Chain by Deep Learning", in Proc. ASRU, 2017]

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202031

Machine Speech Chain▪ Proposed Method→ Develop a closed-loop speech chain model based on deep learning→ The first deep learning model that integrates human speech perception & production behaviors

Not only has the capability to listen and speak,
but also listen while speaking“Goodafternoon”
Sensory
nervesAuditory feedback“How are you?”

Speaking
MotornervesAuditory feedback
Speaking

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202032

Machine Speech Chain
Speaking

Speaking

Feedback
Listening

Listening

A closed-loop architecture:→ In training stage:▪ Allow to train with labeled and unlabeled data (semi-supervised learning)▪ Allow ASR and TTS to teach each other using unlabel data and generate useful feedback→ In Inference stage: Possible to use ASR & TTS module independently
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202033

Overall Architecture
Speaking

Feedback
Listening

Definition:• 𝑥 = original speech, 𝑦 = original text
• 𝑥ො = predicted speech, 𝑦ො = predicted text
• 𝐴𝑆𝑅(𝑥): 𝑥 → 𝑦ො (seq2seq model transform speech to text)• 𝑇𝑇𝑆 𝑦 : 𝑦 → 𝑥ො (seq2seq model transform text to speech)
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202034

Learning in Machine Speech Chain
Case #1: Supervised Learning with Speech-Text Data
Given a pair speech-text 𝒙𝑷 , 𝒚𝑷
• Train ASR and TTS in supervised learning• Directly optimized:→ 𝐴𝑆𝑅 by minimizing ℒ𝑃𝐴𝑆𝑅 𝑦 𝑃 , 𝑦ො 𝑃
→ 𝑇𝑇𝑆 by minimizing ℒ𝑃𝑇𝑇𝑆 𝑥 𝑃 , 𝑥ො 𝑃
• Update both ASR and TTS independently

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202035

Learning in Machine Speech Chain
Case #2: Unsupervised Learning with Speech Only
Given the unlabeled speech features 𝒙𝑼
1. ASR predicts the transcription 𝑦ො 𝑈
2. Based on 𝑦ො 𝑈 , TTS tries to reconstructspeech features 𝑥ො 𝑈
3. Calculate ℒ𝑈𝑇𝑇𝑆 𝑥 𝑈 , 𝑥ො 𝑈 between original
speech features 𝑥 𝑈 and the predicted 𝑥ො 𝑈
Possible to improve TTS with speech onlyby the support of ASR

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202036

Learning in Machine Speech Chain
Case #3: Unsupervised Learning with Text Only
Given the unlabeled text features 𝒚𝑼
1. TTS generates speech features 𝑥ො 𝑈
2. Based on 𝑥ො 𝑈 , ASR tries to reconstructtext features 𝑦ො 𝑈
3. Calculate ℒ𝑈𝐴𝑆𝑅 𝑦 𝑈 , 𝑦ො 𝑈 between originaltext features 𝑦 𝑈 and the predicted 𝑦ො 𝑈
Possible to improve ASR with text onlyby the support of TTS

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202037

Learning in Machine Speech Chain▪ Training Objectiveℒ = 𝛼 ∗ ℒ𝑃𝐴𝑆𝑅 + ℒ𝑃𝑇𝑇𝑆 + 𝛽 ∗ (ℒ𝑈𝐴𝑆𝑅 + ℒ𝑈𝑇𝑇𝑆 )▪ Basic Idea
→ Possible to train the new matters without forgetting the old one→ 𝛼 > 0: keep use some portions of the loss and the gradient provided
by the paired training set→ 𝛼 = 0: completely learn new matters with only speech or only text

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202038

Sequence-to-Sequence ASR
Input & output• 𝒙 = [𝑥1 , … , 𝑥𝑆 ] → speech feature• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] → text

Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑡𝑑 = decoder state at time 𝑡• 𝑎𝑡 = attention probability at time t• 𝑎𝑡 𝑠 = 𝐴𝑙𝑖𝑔𝑛 ℎ𝑠𝑒 , ℎ𝑡𝑑••

𝑎𝑡 𝑠 =

exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠𝑒 ,ℎ𝑡𝑑𝑒 𝑑σ𝑆
𝑠=1 exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠 ,ℎ𝑡
𝑐𝑡 = σ𝑆𝑠=1 𝑎𝑡 𝑠 ∗ ℎ𝑠𝑒 (expected context)

Loss function
𝑇ℒ𝐴𝑆𝑅 𝑦, 𝑝𝑦

1= − ෍ ෍ 1(𝑦𝑡 = 𝑐) ∗ log 𝑝𝑦𝑡 [𝑐]𝑇
𝑡=1 𝑐∈[1..𝐶]

Similar to [LAS, Chan et al. 2015]
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202039

Sequence-to-Sequence TTS
mel-to-linear
end-of-speechprediction

Input & output• 𝒙𝑹 = [𝑥1 , … , 𝑥𝑆 ] (linear spectrogram feature)• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] (mel spectrogram feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)

Loss function
1 𝑆
Reconst. MSE ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො = ෍
𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2
𝑆
𝑠=1
EOS cross entropy
𝑆
1෠ℒ 𝑇𝑇𝑆2 𝑏, 𝑏 = − ෍𝑏𝑠 log(𝑏෠𝑠 ) + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠
𝑆𝑠=1ℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠

Similar to [Tacotron: Wang et al., 2017]

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202040

Experiments on Single-Speaker Speech Chain▪ Features
Speech:• 80 Mel-spectrogram (used by ASR & TTS)
• 1024-dim linear magnitude spectrogram (SFFT) (used by TTS)
• TTS reconstruct speech waveform by using Griffin-Lim
to predict the phase & inverse STFT
Text:
Character-based prediction• a-z (26 alphabet)• 6 punctuation mark (,:’?.-)• 3 special tags 
 
 <spc> (start, end, space)

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202041

Experiments on Single-Speaker Speech Chain▪ Data set→ Single speaker LJSpeech (13,100 utterances)→ Randomly select 94% (total 12,314 utts) for training
3% (total 393 utts) for dev set
3% (total 393 utts) for test set▪ Evaluation→ ASR: Character error rate (CER)→ TTS: L2-norm squared between
the predicted and ground truth
log Mel-spectrogram

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

42

ASR and TTS Results
▪ ASR

▪ TTS

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

43

TTS Subjective Evaluation▪ Mean Opinion Score (MOS)

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202044

Discussion▪ Summary:• Inspired by human speech chain, we proposed machine speech chain to achievesemi-supervised learning• Enables ASR & TTS to assist each other when they receive unpaired data
• Allows ASR & TTS to infer the missing pair and optimize the models with reconstruction loss

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202045

Part II
Multi-speaker Machine Speech Chain[A. Tjandra, S. Sakti, S. Nakamura, "Machine Speech Chain with
One-shot Speaker Adaptation", in Proc. INTERSPEECH, 2018]

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

46

Multi-Speaker Machine Speech Chain▪ Motivation→ Basic Machine Speech Chain was able to improve single-speaker result significantly→ Limitation: couldn’t perform on unseen speaker▪ Proposed Approach: Handle voice characteristics from unknown speakers→ Integrate a speaker recognition system into the speech chain loop
→ Extend the capability of TTS to handle the unseen speaker using one-shot speaker adaptation

Utilizing [Deep speaker; Li et al., 2017]
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202047

Multi-Speaker Machine Speech Chain▪ Train with Speech only:
ASR→TTS𝑥ො =▪ Train with Text only:
TTS→ASR𝑦ො = “text” ℒ𝐴𝑆𝑅 𝑦, 𝑦ොℒ 𝑇𝑇𝑆 𝑥, 𝑥ො

ASR

TTS
𝑧=
SPKREC𝑥ො =𝑦ො = “text”
𝑧ǁ =

ASR

TTS

SPKREC𝑥~
෤ 𝐷𝑃 ∪ 𝐷𝑈

𝑥=→ ASR predicts most possible transcription 𝑦ො→ SPKREC provide a speaker embedding 𝑧→ TTS based on [𝑦,ො z] tries to reconstruct speech 𝑥ො

𝑦= “text”→ Sample a speaker vector 𝑧ǁ from available speech→ TTS generates speech features 𝑥ො based on 𝑦, 𝑧ǁ→ ASR given 𝑥ො tries to reconstruct text 𝑦ොSakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

48

Sequence-to-Sequence TTS
mel-to-linear

end-of-speechprediction

Input & output• 𝒙𝑹 = 𝑥1 , … , 𝑥𝑆 → linear spectrogram• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] → mel spectrogram• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] → text
• 𝒛 → speaker embedding vector
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)
Loss function
1 𝑆
Reconst. MSE ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො = ෍
𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2
𝑆
𝑠=1
෠EOS cross entropy ℒ 𝑇𝑇𝑆2 𝑏, 𝑏
1 𝑆=− ෍𝑏𝑠 log 𝑏෠𝑠 + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠𝑆𝑠=1
Perceptual loss
𝑧,𝑧Ƹ
ℒ
𝑧,
𝑧
Ƹ
=
1
−
(original vs gen sp) 𝑇𝑇𝑆3Ƹ𝑧 2+ 𝑧 2ℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠ + ℒ 𝑇𝑇𝑆3 (𝑧, 𝑧)Ƹ
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

49

Experiments on Multi-Speakers▪ Data set• Training set: Supervised (paired text & speech)• WSJ SI-84 dataset (baseline)(7138 utterances, ~16 h, 84 speakers)• WSJ SI-284 dataset (upperbound)(37318 utterances, ~81 h, 284 speakers)• Training set: Unsupervised (unpaired text & speech)• WSJ SI-200 dataset(30180 utterances, ~66 hours, 200 speakers)• Notes: SI-200 doesn’t overlap with SI-84• Development set: dev93• Evaluation set: eval92
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

50

ASR and TTS Results
▪ ASR

▪ TTS

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

51

TTS Speech Output▪ Text: “the busses aren’t the problem, they actually provide a solution”• Single Speaker (LJSpeech) (p = paired, u = unpaired)
Baseline (P 30%)

Sp-Chain (S 30% + U 70%)

Full (P 100%)• Multispeaker (WSJ)
SpeakerBaseline (P SI84)

Sp-Chain (P si84 + U si200)

Full (P si284)

FemaleA
MaleB

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202052

Discussion▪ Summary:• Improved machine speech chain to handle voice characteristics from unknown speakers→ TTS can generate speech with similar voice characteristic only with one-shot speaker example→ ASR also get new data from the combination between a text sentence and an arbitrary voice
characteristic• By combining both models, we could train with auxiliary feedback loss

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202053

Part III
Cross-Lingual Machine Speech Chain[S. Novitasari, A. Tjandra, S. Sakti, S. Nakamura, "Cross-Lingual Machine Speech Chain forJavanese, Sundanese, Balinese, and Bataks Speech Recognition and Synthesis", in Proc. SLTU, 2020]

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

54

Cross-Lingual Machine Speech Chain▪ Motivation→ Development of ASR and TTS for under-resourced languages are difficult→ A large amount of parallel speech-text data is often unavailable→ The human can learn a new language directly (without textbook) by listening and speaking
Could you repeat after me?

Let's practise your English!

Uhm, yes, my name is …

▪ Proposed Approach: Learn new languages with Machine Speech Chain→ Listening while speaking on new languages→ Enable to perform cross-lingual semi-supervised learning→ No need parallel speech & text of the new language
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

55

Proposed Approach▪ Application: Cross-Lingual Machine Speech Chain forJavanese, Sundanese, Balinese, and Bataks ASR and TTS→ Indonesia is an archipelago comprising approximately 17500 islands→ Approximately, there are 300 ethnic groups, that speak 726 native languages→ Most of them are under-resourced languages
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

56

Learning Process
Step 1: ASR and TTS supervised training using paired speech and text ofrich-resourced language (Indonesian)

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202057

Learning Process
Step 2: ASR and TTS unsupervised training using unpaired data of under-resourced languages(Indonesian ethnic languages: Javanese, Sundanese, Balinese, Bataks)

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202058

Experiments on Cross-lingual Speech Chain▪ Data set• Rich-resourced language (Indonesian language)
Supervised (paired text & speech)→ Full set: 400 spkrs, 84k utterances (~80 hours of speech)→ Test set: 10% of data (40 spkrs)
Remaining data with 360 spkrs (20% dev set; 80% training set)

• Under-resourced language (Ethnics language)
Unsupervised (unpaired data: only text / only speech)→ Full set: 40 spkrs (10 spkrs/languange), 325 utterances/language→ Test set: 10% of data -- 16 spkrs (4 spkrs/language), 50 utterances/language
Remaining data with 36 spkrs (4 spkrs/language), 225 utterances/language(10% dev set; 90% training set)
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

59

ASR and TTS Results
▪ ASR

▪ TTS

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

60

Discussion▪ Summary:• Construct ASR and TTS for ethnic languages (Javanese, Sundanese, Balinese, and Bataks,
when no paired speech or text data were available.
• Pre-trained on Indonesian with parallel speech-text in a supervised manner• Performed speech chain mechanism with only limited text or speech of ethnic languages(unsupervised learning)• Enables ASR and TTS to teach each other even without any paired data

• The framework can be applied to any cross-lingual tasks without significant modification

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

61

Machine Speech Chain Publications
General Machine Speech Chain Framework▪ A. Tjandra, S. Sakti, S. Nakamura, "Listening while Speaking: Speech Chain by Deep Learning", in Proc. IEEE Automatic Speech Recognition andUnderstanding (ASRU) Workshop, 2017
▪ A. Tjandra, S. Sakti, S. Nakamura, "Machine Speech Chain with One-shot Speaker Adaptation", in Proc. INTERSPEECH, 2018
▪ A. Tjandra, S. Sakti, S. Nakamura, "End-to-end Feedback Loss in Speech Chain Framework via Straight-through Estimator", in Proc. IEEE ICASSP,
2019▪ A. Tjandra, S. Sakti, S. Nakamura, "Machine Speech Chain," IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), Vol. 28,
pp. 976-989, 2020

Multilingual Machine Speech Chain▪ S. Nakayama, A. Tjandra, S. Sakti, S. Nakamura, "Speech Chain for Semi-Supervised Learning of Japanese-English Code-Switching ASR and TTS",in Proc. SLT, 2018
▪ S. Nakayama, A. Tjandra, S. Sakti, S. Nakamura, "Zero-shot Code-switching ASR and TTS with Multilingual Machine Speech Chain,“ in Proc. IEEE
Automatic Speech Recognition and Understanding (ASRU) Workshop, 2019
▪ S. Novitasari, A. Tjandra, S. Sakti, S. Nakamura, "Cross-Lingual Machine Speech Chain for Javanese, Sundanese, Balinese, and Bataks Speech
Recognition and Synthesis", in Proc. SLTU, 2020

Multimodal Machine Speech Chain▪ J. Effendi, A. Tjandra, S. Sakti, S. Nakamura, “Listening while Speaking and Visualizing: Improving ASR through Multimodal Chain,” in Proc. IEEE
Automatic Speech Recognition and Understanding (ASRU) Workshop, 2019
▪ J. Effendi, A. Tjandra, S. Sakti, S. Nakamura, "Augmenting Images for ASR and TTS through Single-loop and Dual-loop Multimodal Chain
Framework," in Proc. of INTERSPEECH, pp. to appear, 2020

Incremental (Real-time) Machine Speech Chain▪ S. Novitasari, A. Tjandra, T. Yanagita, S. Sakti, S. Nakamura, "Incremental Machine Speech Chain for Enabling Listening while Speaking in Realtime," in Proc. of INTERSPEECH, pp. to appear, 2020
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 2020

62

Citations▪ [Denes & Pinson, 1993] -- P. Denes and E. Pinson, “The Speech Chain”, ser. Anchor books. Worth Publishers, 1993. [Online]. Available:
https://books.google.co.jp/books? id=ZMTm3nlDfroC▪ [Bosi & Goldberg, 2003] – M. Bosi, and R.E. Goldberg, “Introduction to digital audio coding and standards”, Boston: Kluwer Academic Pub., 2003
▪ [Zen et al., 2009] -- H. Zen, K. Tokuda, and A. Black, “Statistical parametric speech synthesis,” Speech Comm., vol. 51, no. 11, pp. 1039–1064, 2009
▪ [Sakti et al., 2008] -- Sakti, S., Kelana, E., Riza, H., Sakai, S., Markov, K., Nakamura, S., 2008. Development of Indonesian LVCSR system within ASTAR project. In: Proc. Workshop on Technologies and Corpora for Asia-Pacific Speech Translation, Hyderabad, India, pp. 19–24.
▪ [Sakti et al., 2013] -- S. Sakti, M. Paul, A. Finch, S. Sakai, T.-T. Vu, N. Kimura, C. Hori, E. Sumita, S. Nakamura, J. Park, C. Wutiwiwatchai, B. Xu, H.
Riza, K. Arora, C.-M. Luong, H. Li, "A-STAR: Toward Translating Asian Spoken Languages", Special issue on S2ST, Computer Speech andLanguage Journal (Elsevier), vol. 27, Issue 2, pp. 509-527, February 2013
▪ [Sakti et al., 2015] -- S. Sakti, O. Shagdar, F. Nashashibi, S. Nakamura, "Context awareness and priority control for ITS based on automatic speech
recognition", in Proc. 14th International Conference on ITS Telecommunications, 2015
▪ [Xiong et al., 2017] -- W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu and G. Zweig, "Achieving Human Parity inConversational Speech Recognition“, Microsoft Research Technical Report MSR-TR-2016-71, 2017
▪ [Saon et al., 2017] -- G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L. Lim, B. Roomi,
P. Hall, "English Conversational Telephone Speech Recognition by Humans and Machines“, ASRU 2017
▪ [Waldstein, 1990] – R.S. Waldstein, “Effects of postlingual deafness on speech production: Implications for the role of auditory feedback. J. Acoust.
Soc. Am. 88, 2099–2114, 1990
▪ [Hickok, 2003] – G. Hickok and B. Buchsbaum, “Temporal lobe speech perception systems are part of the verbal working memory circuit: Evidence
from two recent fMRI studies. Behav. Brain Sci. 26, 740–741, 2003
▪ [Hickok, 2011] – G. Hickok, J. Houde, F. Rong, "Sensorimotor Integration in Speech Processing: Computational Basis and Neural Organization",▪ Neuron Perspective, Vol. 69, Issue 3, pp. 407-422, 2011
▪ [Tjandra, Sakti, and Nakamura, ASRU 2017a] -- A. Tjandra, S. Sakti, S. Nakamura, "Attention-based Wav2Text with Feature Transfer Learning", inProc. ASRU, 2017
▪ [Tjandra, Sakti, and Nakamura, ASRU 2017b] -- A. Tjandra, S. Sakti, S. Nakamura, "Listening while Speaking: Speech Chain by Deep Learning", inProc. ASRU, 2017
▪ [Tjandra, Sakti, and Nakamura, INTERSPEECH 2018] -- A. Tjandra, S. Sakti, S. Nakamura, "Machine Speech Chain with One-shot Speaker
Adaptation", in Proc. INTERSPEECH, 2018
Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202063

Thank you

Sakriani Sakti @ AHC Labs, NAIST/RIKEN AIP, Japan | AIST Workshop (India) | August 20th, 202057