Towards Developing
Neural Machine Speech Interpreterthat Listens, Speaks, and Listens while Speaking
Sakriani Sakti
Joint work with:
Andros Tjandra, Sashi Novitasari, Tomoya Yanagita,
Sahoko Nakayama, Johanes Effendi, and Satoshi Nakamura
1Nara Institute of Science and Technology (NAIST)
2RIKEN Advanced Intelligence Project (RIKEN AIP)

1

What is Speech Interpreter?

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 20202

Professional Speech Interpreter

Speaker
Nice to meet you. I am interested in doing business together.
Interpreter
はじめまして。一緒にビジネスをすることに興味があります。
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 20203

Speech-to-Speech Translation
Speech-to-Speech Translation(S2ST)
Japanese「私は学校に行く」

English“I go to school”
SpeechRecognition(ASR)

I go toschoolMachineTranslation
(MT)

私は
学校に行く

SpeechSynthesis(TTS)

Focus on speech language technologies for S2ST
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 20204

Automatic Speech Recognition (ASR)▪ Early Technologies for ASR→ Template Matching, Dynamic Programing [Sakoe et al., 1971]→ Hidden Markov Modeling [Baum et al., 1966]→ Neural Network, TDNN [Waibel et al. 1989], LSTM [Hochreiter et al., 1997]→ Weighted Finite State Transducer [Mohri et al., 2002]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 20205

Automatic Speech Recognition (ASR)▪ Recent Technologies for ASR→ Hybrid HMM-DNN [Borlard et al., 1993]
Estimate State Posterior Probability
by DNN→ Connectionist Temporal Classification[Graves et al., 2006]
Predict Phoneme Label every frame→ Listen, Attend, and Spell [Chan et al., 2016]
Sequence-to-sequence modeling▪ Important Factors of
Deep Learning→ Simplify many complicatedhand-engineered models→ Let the networks find the waythat map from speech to text

[LAS, Chan et al. 2015;
Figure courtesy of A. Tjandra]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

6

ASR Progress[Xiaong et al., 2017][Saon et al., 2017][https://www.economist.com/technology-quarterly/2017-05-01/language]

IBM vs. Microsoft: “Human parity” speech recognition record
Makes the same / fewer errors than professional transcriptionists
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 20207

Text-to-Speech Synthesis (TTS)▪ From hidden Markov Model (HMM) to Deep Learning• Conditional WaveNet – TTS[Wavenet; Oord et al., 2016][Zen et al., 2009][Tacotron; Wang et al., 2017;
Figure courtesy of A. Tjandra]
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

8

TTS Performance▪ From robot voice to human-like voice[Source: https://www.economist.com/technology-quarterly/2017-05-01/language]▪ Google Duplex
Duplex scheduling a hair salon appointment:

Duplex calling a restaurant:[Source: https://ai.googleblog.com/2018/05/
duplex-ai-system-for-natural-conversation.html]
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 20209

Have we solved all problems?

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202010

Professional Speech Interpreter
Speak
Listen→ The translation process starts before
receiving the end of the sentence→ Has the ability to do simultaneous
process

Challenges formachine speech interpreter:
SpeakerInterpreter

1. Requires the ability to listen
while speaking
2. Requires the ability to perform
recognition and synthesis speechin real-time

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202011

Approach to Problem 1

Listens while Speaking:
Machine Speech Chain by Deep Learning

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202012

Human Communication▪ Speech Chain [Denes & Pinson, 1993]“Good
afternoon”

Sensory
nervesAuditory feedback
Listening

Speaking
Motornerves

Linguistic
Level

Physiological
Level

Acoustic
Level

Physiological
Level

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

Sensory
nerves

Linguistic
Level

13

Human Communication▪ Human: Learning to Listen and Speak→ Humans learn how to talk by constantly repeating their articulations & listening to sounds produced→ A closed-loop speech chain mechanism has critical auditory feedbackChildren who lose their hearing often
have difficulty in producing clear speech“Goodafternoon”
Sensory
nervesAuditory feedback

Adults who become deaf
after becoming proficient with a language
nonetheless suffer speech articulation declines
as a result of the lack of auditory feedback

Speaking[Waldstein, 1990]

MotornervesSakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202014

Human Communication▪ Human Brain: Sensorimotor Integration in Speech Processing(1) the auditory system is critically involved in the perception of speech(2) the motor system is critically involved in the production of speech

Spt exhibits sensorimotor response properties, activating
both during the passive perception of speech and during
covert (subvocal) speech articulation [Hickok et al, 2003]

An Integrated State Feedback Control (SFC) Modelof Speech Production [Hickok et al. 2011]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202015

Human-Machine Communication▪ Machine: Learning to Listen and Speak→ Computers are able to learn how to listen or learn how to speak→ But, computers cannot hear their own voice→ The learning to listen and speak is done separately and independently→ Requires a lot of parallel speech and text to train in a supervised way (more than human need)

Listening“Good afternoon”“How are you?”

Recognized words

Speech recognition→ Only listening
Speaking

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

Speech synthesis→ Only speaking

16

Part 1-1
Basic Machine Speech Chain[A. Tjandra, S. Sakti, S. Nakamura, "Listening while Speaking:Speech Chain by Deep Learning", in Proc. ASRU, 2017]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202017

Machine Speech Chain▪ Proposed Method→ Develop a closed-loop speech chain model based on deep learning→ The first deep learning model that integrates human speech perception & production behaviors

Not only has the capability to listen and speak,
but also listen while speaking“Goodafternoon”
Sensory
nervesAuditory feedback“How are you?”

Speaking
MotornervesAuditory feedback
Speaking

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202018

Machine Speech Chain
Speaking

Speaking

Feedback
Listening

Listening

Advantages of closed-loop architecture:→ In training stage:▪ Allow to train with labeled and unlabeled data (semi-supervised learning)▪ Allow ASR and TTS to teach each other using unlabeled data and generate useful feedback→ In Inference stage: Possible to use ASR & TTS module independently
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

19

Overall Architecture
Speaking

Feedback
Listening

Definition:• 𝑥 = original speech, 𝑦 = original text
• 𝑥ො = predicted speech, 𝑦ො = predicted text
• 𝐴𝑆𝑅(𝑥): 𝑥 → 𝑦ො (seq2seq model transform speech to text)• 𝑇𝑇𝑆 𝑦 : 𝑦 → 𝑥ො (seq2seq model transform text to speech)
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202020

Learning in Machine Speech Chain
Case #1: Supervised Learning with Speech-Text Data
Given a pair speech-text 𝒙𝑷 , 𝒚𝑷
• Train ASR and TTS in supervised learning• Directly optimize:→ 𝐴𝑆𝑅 by minimizing ℒ𝑃𝐴𝑆𝑅 𝑦 𝑃 , 𝑦ො 𝑃
→ 𝑇𝑇𝑆 by minimizing ℒ𝑃𝑇𝑇𝑆 𝑥 𝑃 , 𝑥ො 𝑃
• Update both ASR and TTS independently

𝑦 𝑃 = “texts”

𝑥𝑃 =ℒ𝑃𝐴𝑆𝑅 𝑦 𝑃 , 𝑦ො 𝑃

ℒ𝑃𝐴𝑆𝑅 𝑥 𝑃 , 𝑥ො 𝑃

𝑦ො 𝑃 = “text”𝑥ො 𝑃 =ASR

𝑥𝑃 =Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

TTS
𝑦 𝑃 = “text”

21

Learning in Machine Speech Chain
Case #2: Unsupervised Learning with Speech Only
Given only speech features 𝒙𝑼
1. ASR predicts the most possible
transcription𝑦ො 𝑈
2. Based on 𝑦ො 𝑈 , TTS tries to reconstructspeech features 𝑥ො 𝑈
𝑇𝑇𝑆 𝑈 𝑈
3. Calculate ℒ𝑈
𝑥 , 𝑥ො between original
speech features 𝑥 𝑈 and the predicted 𝑥ො 𝑈
Possible to improve TTS with speech onlyby the support of ASR

𝑥ො 𝑈 =ℒ𝑈𝑇𝑇𝑆 𝑥 𝑈 , 𝑥ො 𝑈

TTS
𝑦ො 𝑈 = “text”
ASR
𝑥𝑈 =

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202022

Learning in Machine Speech Chain
Case #3: Unsupervised Learning with Text Only
Given only text features 𝒚𝑼
1. TTS generates speech features 𝑥ො 𝑈
2. Based on 𝑥ො 𝑈 , ASR tries to reconstructtext features 𝑦ො 𝑈
3. Calculate ℒ𝑈𝐴𝑆𝑅 𝑦 𝑈 , 𝑦ො 𝑈 between originaltext features 𝑦 𝑈 and the predicted 𝑦ො 𝑈

𝑦ො 𝑈 = “text”ℒ𝑈𝐴𝑆𝑅 𝑦 𝑈 , 𝑦ො 𝑈

ASR
𝑥ො 𝑈 =
TTS

Possible to improve ASR with text onlyby the support of TTS

𝑦 𝑈 = “text”

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202023

Learning in Machine Speech Chain▪ Training Objectiveℒ = 𝛼 ∗ ℒ𝑃𝐴𝑆𝑅 + ℒ𝑃𝑇𝑇𝑆 + 𝛽 ∗ (ℒ𝑈𝐴𝑆𝑅 + ℒ𝑈𝑇𝑇𝑆 )▪ Basic Idea
→ Possible to train the new matters without forgetting the old one→ 𝛼 > 0: keep using some portions of the loss and the gradient provided
by the paired training set→ 𝛼 = 0: completely learn new matters with only speech or only text

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202024

Sequence-to-Sequence ASR
Input & output• 𝒙 = [𝑥1 , … , 𝑥𝑆 ] → speech feature• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] → text
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑡𝑑 = decoder state at time 𝑡• 𝑎𝑡 = attention probability at time t• 𝑎𝑡 𝑠 = 𝐴𝑙𝑖𝑔𝑛 ℎ𝑠𝑒 , ℎ𝑡𝑑••

𝑎𝑡 𝑠 =

exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠𝑒 ,ℎ𝑡𝑑𝑒 𝑑σ𝑆
𝑠=1 exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠 ,ℎ𝑡
𝑐𝑡 = σ𝑆𝑠=1 𝑎𝑡 𝑠 ∗ ℎ𝑠𝑒 (expected context)

Loss function
𝑇ℒ𝐴𝑆𝑅 𝑦, 𝑝𝑦

1= − ෍ ෍ 1(𝑦𝑡 = 𝑐) ∗ log 𝑝𝑦𝑡 [𝑐]𝑇
𝑡=1 𝑐∈[1..𝐶]

Similar to [LAS, Chan et al. 2015]
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202025

Sequence-to-Sequence TTS
mel-to-linear
end-of-speechprediction

Input & output• 𝒙𝑹 = [𝑥1 , … , 𝑥𝑆 ] (linear spectrogram feature)• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] (mel spectrogram feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)
Loss function

1 𝑆
Reconst. MSE ℒ
ො = ෍
𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2
𝑇𝑇𝑆1 𝑥, 𝑥
𝑆
𝑠=1
EOS cross entropy
𝑆
1෠ℒ 𝑇𝑇𝑆2 𝑏, 𝑏 = − ෍𝑏𝑠 log(𝑏෠𝑠 ) + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠
𝑆𝑠=1ℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠

Similar to [Tacotron: Wang et al., 2017]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202026

Experiments on Single-Speaker Dataset▪ Features
Speech:• 80-dim Mel-spectrogram (used by ASR & TTS)
• 1024-dim linear magnitude spectrogram (SFFT) (used by TTS)
• TTS reconstruct speech waveform by using Griffin-Lim
to predict the phase & inverse STFT
Text:
Character-based prediction• a-z (26 alphabet)• 6 punctuation mark (,:’?.-)• 3 special tags 
 
 <spc> (start, end, space)

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202027

Experiments on Single-Speaker Dataset▪ Data set• Natural speech single-speaker dataset: LJSpeech [Ito et al. , 2017]• Training set: 12,314 utts; dev set: 393 utts; test set:393 utts

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202028

Experiments on Single-Speaker Dataset▪ ASR results

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202029

Experiments on Single-Speaker Dataset▪ TTS results

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202030

Discussion▪ Summary:• Inspired by the human speech chain, we proposed a machine speech chain to achievesemi-supervised learning• Enables ASR & TTS to assist each other when they receive unpaired data
• Allows ASR & TTS to infer the missing pair and optimize the models with reconstruction losses▪ Current Limitations:• Set of speakers is fixed → Unable to handle unseen speakers• TTS system only mimics the voices of diff. speakers via speaker’s identity by one-hot encoding• ASR only adapted to a specific set of speaker→ Because the TTS unable to produce more voice characteristics from unseen speakers

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202031Part 1-2
Multi-Speaker Machine Speech Chain[A. Tjandra, S. Sakti, S. Nakamura, "Machine Speech Chain with
One-shot Speaker Adaptation", in Proc. INTERSPEECH, 2018]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202032

Multi-Speaker Machine Speech Chain▪ Proposed Approach: Handle voice characteristics from unknown speakers→ Integrate a speaker recognition system into the speech chain loop
→ Extend the capability of TTS to handle the unseen speaker using one-shot speaker adaptation→ Coupling with ASR, we developed a speech chain framework that is able to adapt
new data speech from an unknown speaker
𝑥ො =

TTS

𝑥ො =

𝑦 = “text”𝑦ො = “text”

𝑧=
SPKREC𝑥=ASR

𝑦ො = “text”

Utilizing [Deep speaker; Li et al., 2017]
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202033

Learning in Multi-Speaker Speech Chain▪ Train with Speech only:
ASR→TTS𝑥ො =▪ Train with Text only:
TTS→ASR𝑦ො = “text”ℒ 𝑇𝑇𝑆 𝑥, 𝑥ො

ASR

TTS

𝑧=
SPKRECℒ𝐴𝑆𝑅 𝑦, 𝑦ො𝑥ො =𝑦ො = “text”
𝑧ǁ =

ASR

TTS

SPKREC𝑥~
෤ 𝐷𝑃 ∪ 𝐷𝑈

𝑥=→ ASR predicts most possible transcription 𝑦ො→ SPKREC provides a speaker embedding 𝑧→ Based on [𝑦,ො z], TTS tries to reconstruct speech 𝑥ො

𝑦= “text”→ Sample a speaker vector 𝑧ǁ from available speech→ TTS generates speech features 𝑥ො based on 𝑦, 𝑧ǁ→ Given 𝑥,
ො ASR tries to reconstruct text 𝑦ොSakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202034

Sequence-to-Sequence TTS
mel-to-linear

end-of-speechprediction

Input & output• 𝒙𝑹 = 𝑥1 , … , 𝑥𝑆 → linear spectrogram• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] → mel spectrogram• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] → text
• 𝒛 → speaker embedding vector
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)
Loss function
1 𝑆
Reconst. MSE ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො = ෍
𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2
𝑆
𝑠=1
෠EOS cross entropy ℒ 𝑇𝑇𝑆2 𝑏, 𝑏
1 𝑆=− ෍𝑏𝑠 log 𝑏෠𝑠 + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠𝑆𝑠=1
Perceptual loss
𝑧,𝑧Ƹ
ℒ
𝑧,
𝑧
Ƹ
=
1
−
(original vs gen sp) 𝑇𝑇𝑆3Ƹ𝑧 2+ 𝑧 2ℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠ + ℒ 𝑇𝑇𝑆3 (𝑧, 𝑧)Ƹ
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202035

Experiments on Multi-Speakers▪ Data set: Wall Street Journal (WSJ) [Paul et al. , 1992]• Training set: Supervised (paired text & speech)• WSJ SI-84 dataset (baseline)(7138 utterances, ~16 h, 84 speakers)• WSJ SI-284 dataset (upperbound)(37318 utterances, ~81 h, 284 speakers)• Training set: Unsupervised (unpaired text & speech)• WSJ SI-200 dataset(30180 utterances, ~66 hours, 200 speakers)• Notes: SI-200 doesn’t overlap with SI-84• Development set: dev93• Evaluation set: eval92
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

36

ASR Results
Model
CER (%)
Supervised training:
WSJ train_si84 (16hrs speech, paired) -> Baseline
Att Enc-Dec
17.35

Supervised training:
WSJ train_si284 (81 hrs speech, paired) -> UpperboundAtt Enc-Dec
7.12
Semi-supervised training:
WSJ train_si84 (paired) + train_si200 (unpaired)Label propagation (greedy)
17.52
Label propagation (beam=5)
14.58
Proposed speech chain
9.86
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

37

TTS Results▪ Text: “The busses aren’t the problem, they actually provide a solution.”• Single Speaker (LJSpeech) (p = paired, u = unpaired)
Baseline (P 30%)

Sp-Chain (S 30% + U 70%)

Full (P 100%)• Multispeaker (WSJ)
SpeakerBaseline (P SI84)

Sp-Chain (P si84 + U si200)

Full (P si284)

FemaleA
MaleB

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202038

Discussion▪ Summary:• Improved machine speech chain to handle voice characteristics from unknown speakers→ TTS can generate speech with similar voice characteristic only with one-shot speaker examples→ ASR also get new data from the combination between a text sentence and an arbitrary voice
characteristic• By combining both models, we could train with auxiliary feedback loss

▪ Current Limitations:• If we only have the text: Perform TTS →ASR, and update only ASR with feedback loss
• If we only have speech: Perform ASR →TTS, and update only TTS with feedback loss
• Backpropagation the error from the reconstruction loss through ASR is challenging
due to the output of the ASR discrete tokens

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202039

Part 1-3
Machine Speech Chainwith End-to-end Feedback Loss
[A. Tjandra, S. Sakti, S. Nakamura, "End-to-end Feedback Loss in Speech
Chain Framework via Straight-through Estimator", in Proc. ICASSP, 2019]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202040

Straight-Through Estimator for Speech Chain▪ Proposed Approach: Handle backpropagation through discrete nodes

Feedback loss: ℒ 𝑇𝑇𝑆 (𝑥, 𝑥)
ො where 𝑥 = 𝑇𝑇𝑆(𝑦,ො 𝑧)a) Speech chain loop with speaker embedding module 𝑧b) Original: feedback ℒ 𝑇𝑇𝑆 can’t be backpropagated through variable 𝑦ොc) Proposal: Estimate gradient through variable 𝑦ො with straight-through estimator
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202041

Basic Idea
▪ Based on Two ICLR 2017 Papers [Jang et al. 2017, Maddison et al. 2017]▪ Example: Discrete Node with Step Function
Gumbel-Softmax Distribution
Provides a simple method for draw samples
from a categorical distribution with class
Probabilities→ Use softmax function as approximate ofargmax function
Almost everywhere, a small change ininput results in no change in output→ Gradient is zero

A common trick is to use acontinuous approximation→ But, they fail to produce
discrete outputs[Source: https://uoguelph-mlrg.github.io/spaceNet_overview2/]
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202042

Straight-Through Estimatora)ST-argmax
Deterministic choosing token by highest probability
𝑝𝑦𝑡 c =

exp ℎ𝑡𝑑 [𝑐]σ𝐶𝑖=1 exp ℎ𝑡𝑑 [𝑐]𝑦෤𝑡 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑐 𝑝𝑦𝑡 [𝑐]b)ST-Gumbel softmax
Sampling a token from 𝑝𝑦𝑡 𝑐 :𝑝𝑦𝑡 𝑐 =

exp (ℎ𝑡𝑑 𝑐 + 𝑔𝑐 )/𝜏

σ𝐶𝑖=1 exp (ℎ𝑡𝑑 𝑐 + 𝑔𝑐 )/𝜏
𝑦෤𝑡 ~𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑖𝑐𝑎𝑙 𝑝𝑦𝑡 1 , . . , 𝑝𝑦𝑡 𝐶𝜏 = temperatureℎ𝑡𝑑 =logit ASR

New gradient ℒ 𝑇𝑇𝑆 w.r.t. 𝜃𝐴𝑆𝑅

Replace gradient from computation𝜕෦𝑦node 𝜕𝑝 𝑡 with identity
𝑦𝑡

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202043

Experiments on Multi-Speakers WSJ Task
▪ Data set• Training set: Supervised (paired text & speech)
WSJ SI-284 dataset [Paul et al. , 1992](37318 utterances, ~81 h, 284 speakers)• Development set: dev93

Model• Evaluation set: eval92

CER (%)

Baseline
Enc-Dec Att-MLP [Kim et al., 2017]

11.08

Enc-Dec Att-MLP-Loc [Kim et al., 2017]

8.17

Enc-Dec Att-MLP [Tjandra et al., 2017]

7.12

Enc-Dec Att-MLP-MA (ours) [Tjandra et al., 2018]

6.43

Proposed Method
Enc-Dec Att-MLP-MA SP-Chain ST argmax

5.75

Enc-Dec Att-MLP-MA SP-Chain ST gumbel

5.70

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202044

Discussion▪ Summary:• Improved machine speech chain mechanism→ Allow backpropagation through discrete output with a straight-through estimator• Future work:
It is necessary to validate the effectiveness of the approach in various languages

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202045

Part 1-5
From Speech Chain to Multimodal Chain[Johanes Effendi, Andros Tjandra, Sakriani Sakti, Satoshi Nakamura, “Augmenting Images for ASR and TTS
through Single-loop and Dual-loop Multimodal Chain Framework,” Proc. of INTERSPEECH, Oct 2020]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202046

Human Communication▪ Human Communication:• The most common way for humans to communicate is by speech• But, a language system cannot know what it is communicating without a connection
to the real world by image perception

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202047

Human Communication▪ Human Communication:• Human communication is multisensory and involves several communication channels(auditory and visual channels)• Human perceives these multiple sources of information together to build a general concept
Auditory:
Listening

Visual: Reading/Watching
Verbal: Speaking

Auditory
feedback

Each modalities shares complementary behaviour

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

48

Multimodal System▪ Multimodal System:• The idea of incorporating visual information for speech processing is not new→ Audio-visual ASR [Petridis et al., 2017, Chung et al., 2017, Afouras et al., 2018]• But most approaches are usually made by simply concatenating the information→ Inefficient to effectively fuse information• These methods require all information from different modalities available altogether→ Parallel data is often unavailable
Image

SpeechSpeechSpeechRecognition

Text

Image

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

49

Machine Speech Chain▪ Machine Speech Chain:• The approach let us free from the need for a large size of parallel speech-text data
• It provides possibilities to improve ASR & TTS performance in semi-supervised learning by
allowing ASR and TTS to teach each other, given only text or only speech data.
Unfortunately,• Although it reduces the requirement of having a full amount of paired data,
it is still required to have a large size of unpaired data
• This study is limited only to speech and textual modalities• Natural communication is multimodal that involves

not only the auditory system but also visual sensory

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

50

Multimodal Machine Speech Chain (MMC)▪ Multimodal Machine Speech Chain (MMC):• Expanding the speech chain (a) into a multimodal chain (b)• Design a closely-knit chain architecture that connects ASR, TTS, IC, and IR/IG• Can be trained in semi-supervised fashion by assisting each other given incomplete data
• Leveraging cross-modal data augmentation within the chain▪ Research Question: Can we still improve ASR even no speech/text data available?
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

51

Learning in MMC Framework
Case #1: Supervised Learning with Speech-Text-Image Data
▪ Paired speech-text-image data exist:→ Separately train ASR, TTS, IC and IR/IG (supervised learning)

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202052

Learning in MMC Framework
Case #2: Unsupervised Learning with Unpaired Data
▪ Unpaired speech-text-image data exist:→ Perform speech chain (ASR-TTS) with unpaired speech-text data
→ Perform image chain (IC-IG) with unpaired image-text data
Given only image or speech data

Speechchain

Visualchain

Given only text data

Speechchain

Visualchain

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202053

Learning in MMC Framework
Case #3: Unsupervised Learning with Speech or Image Only Data
▪ Single data exist (speech/text/image only):→ Only text data: Perform unrolled process(1) TTS→ASR to update TTS&ASR and generate speech data
(2) IG→IC to update IG&IC and generate image data
→ Only speech data:
Unrolled process ASR→TTS to update TTS&ASR and generate text
then use it on IG→IC to update IG&IC and generate image data
→ Only image data:
Unrolled process IC→IG to update IG&IC and generate text,
then use it on TTS→ASR to update TTS&ASR and generate speech
Given only speech data

Given only image data

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

54

Multimodal Machine Speech Chain▪ Multimodal Machine Speech Chain:→ Alternative framework: Simplified single-loop multimodal chain→ Human brain process visual and auditory components of speech in a unified manner[Calvert, 2001]

MMC1-IR/IG - Dual loop multimodal chain(Proposed)

MMC2 – Single loop multimodal chain(Proposed)

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

55

Learning in MMC2 Framework
Case #2: Unsupervised Learning with Unpaired Data

▪ Unpaired speech-text-image data exist:
Given only image and/or speech data

Given only text data

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202056

IC and IG Architecture▪ Image Captioning:

Similar to “Show, attend and tell” [Xu et al. 2015]▪ Image Generation:

Similar to “AttnGAN - Multistep image generation
using adversarial loss” [Xu et al., 2017]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

57

ImgSp2Txt Architecture▪ Image-Speech to Text:→ Encoders have similar settings with ASR & IC
→ Combine output layer of IC and ASR• use combination for the next step• both models trained altogether→ When both image and speech are available:• use average of both output layer→ When only image or speech are available:• use the corresponding modality output layer

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202058

Data Set-Up▪ Data→ Flickr8k [Ratschian et al., 2010]→ 8k images, 5 captions/image→ 65 hours of natural speech
multi-speaker English speech[Harwath and Glass, 2015]Type

Speech

Text

Image

# Image

Multimodal
Paired

○

○

○

800

Multimodal
Unpaired

∆

∆

∆

1500

Speech only

∆

x

x

1850

Image only

x

x

∆

1850

○ : available paired∆ : available but unpaired
x : unavailable
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

59

ASR-TTS Results
Training

Data Type#ImageASR (CER) ↓IC (BLEU4) ↑Multimodal (P)

800

36.35

12.75

0.77

5.90

1500

15.10

13.22

0.59

8.29

1850

12.37

13.28

0.56

9.12

1850

12.06

13.29

0.56

9.11

6000

5.76

19.91

0.50

9.66

800

26.67

32.23

0.77

5.90

1500

14.88

55.15

0.65

10.12

1850

13.81

58.03

0.62

10.65

+ Img only (U)

1850

12.32

59.66

0.61

9.95

Multimodal (P)

6000

5.16

79.88

0.50

9.66

MMC 1
+ Multimodal (U)Dual-loop(Semi-supervised) + Sp only (U)+ Img only (U)Topline MMC1
(Supervised)

Multimodal (P)Multimodal (P)MMC 2+ Multimodal (U)Single-loop(Semi-supervised) + Sp only (U)Topline MMC2(Supervised)

TTS (L22 Norm) ↓ IG (Inception) ↑ASR could still be improved even without speech and text data
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

60

Discussion▪ Summary:• Machine speech chain enables semi-supervised learning without parallel data
• We upgrade the speech chain into the multimodal chain
by jointly training IC and IG model in a loop connection• It allows us to improve ASR even only image data is available

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

61

Professional Speech Interpreter
Speak
Listen→ The translation process starts before
receiving the end of sentence→ Has the ability to do simultaneous
process

Challenges formachine speech interpreter:
SpeakerInterpreter

1. Requires the ability to listen
while speaking
2. Requires the ability to perform
recognition and synthesis speechin real-time

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

62

Approach to Problem 2

Incremental ASR and TTS for
Real-time Machine Speech Interpreter

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202063

Real-time Machine Speech Interpreter▪ Traditional Speech Translation
SpeechRecognition(ASR)

I go toschoolMachineTranslation
(MT)

私は
学校に行く

SpeechSynthesis(TTS)▪ Real-time Machine Speech Interpreter

SpeechRecognition
(ASR)

I go

SpeechRecognition(ASR)

MachineTranslation
(MT)

to
shool

私は

MachineTranslation(MT)

SpeechSynthesis
(TTS)

学校
に行く

SpeechSynthesis(TTS)

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202064

Part 2-1
Neural Incremental Speech Recognition[S. Novitasari, A. Tjandra, S. Sakti, and S. Nakamura, "Sequence-to-Sequence Learning via Attention Transfer
for Incremental Speech Recognition," in Proc. INTERSPEECH, 2019]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202065

Real-time Machine Speech Interpreter▪ Traditional Speech Translation
SpeechRecognition(ASR)

I go toschoolMachineTranslation
(MT)

私は
学校に行く

SpeechSynthesis(TTS)▪ Real-time Machine Speech Interpreter

SpeechRecognition
(ASR)

I go

SpeechRecognition(ASR)

MachineTranslation
(MT)

to
shool

私は

MachineTranslation(MT)

SpeechSynthesis
(TTS)

学校
に行く

SpeechSynthesis(TTS)

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

66

Neural ASR▪ Current Attention-based Seq2Seq Neural ASR→ Standard attention has a “global” property :Attend whole input sequence and calculate the expected context→ The output is generated after receiving the entire input sequence→ Requires a significant delay to recognize long utterances▪ Related Works on Incremental ASR (ISR)→ Local attention mechanism [Bahdanau et al., 2014; Tjandra et al., 2017]:

Limit the area of attention, but without reducing the latency→ Character-level Incremental Speech Recognition with Recurrent Neural Networks[Hwang et al., 2016]- Unidirectional RNN-CTC- Requires depth-pruning in beam search during output generation→ Neural transducer [Jaitly et al., 2016]- Seq2seq model: recognize speech part-by-part- Requires to infer alignment during training- Use dynamic programming to approximate best alignment
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

67

Neural ASR▪ Problems→ Most existing Neural ISR models utilize different frameworks and learning algorithms
that are more complicated than standard Neural ASR→ The models need to decide: (1) Incremental steps(2) Learn the transcription that aligns with the current short speech segment▪ Proposed Neural ISR→ Employing the original architecture of attention-based ASR with shorter sequences→ Perform attention transfer: full-utterance ASR as the teacher model and ISR as the student model→ Mimics the speech-text alignment produced by standard ASR

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

68

Proposed Neural ISR●

In training, if it reaches the last
token aligned to the block, thedecoder will learn to output an<\m> symbol●

In actual use, decoding ineach step stops when:−<\m> predicted, or−
token length. reaches athreshold (max)<\m> → end-of-block●

Notation:x – speech framey – token
w – frame block size
stateℎ𝑒𝑑 –– encoderstateℎ – decoder𝑎 attentionc – attention context
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202069

Experimental Results▪ Data set• Training set: Supervised (paired text & speech)
WSJ SI-284 dataset [Paul et al. , 1992](37318 utterances, ~81 h, 284 speakers)• Development set: dev93• Evaluation set: eval92

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202070

Discussion▪ Summary:• Develop incremental ASR that employ original architecture of neural ASR• Perform transfer learning:- Treat standard ASR as a teacher model and ISR as a student model• Experimental results:- Successfully reduced the delay- Achieved comparable performance than standard ASR that wait until the end

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202071

Part 2-2
Neural Incremental Speech Synthesis[T. Yanagita, S. Sakti, S. Nakamura, "Neural iTTS: Toward Synthesizing Speech in Real-time with
End-to-end Neural Text-to-Speech Framework," in Proc. Speech Synthesis Workshop (SSW), 2019]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202072

Real-time Machine Speech Interpreter▪ Traditional Speech Translation
SpeechRecognition(ASR)

I go toschoolMachineTranslation
(MT)

私は
学校に行く

SpeechSynthesis(TTS)▪ Real-time Machine Speech Interpreter

SpeechRecognition
(ASR)

I go

SpeechRecognition(ASR)

MachineTranslation
(MT)

to
shool

私は

MachineTranslation(MT)

SpeechSynthesis
(TTS)

学校
に行く

SpeechSynthesis(TTS)

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

73

Neural ITTS
▪ Training• Randomly splitting the sentence into shorter parts
: sentence start
: sentence end<m>: middle sentence start</m>: middle sentence end

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

74

Neural ITTS
▪ Architecture• Full-sentence: 
 私は学校に行く 

• ITTS perform incremental on shorter units
私は</m><m>学校に行く

: sentence start
: sentence end<m>: middle sentence start</m>: middle sentence end
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202075

Experiment Set-Up▪ Set-Up:

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202076

Experiment Results: Subjective Evaluation▪ MOS Results:
4.95

5
4
2.81

3

2.88

3.13

3.18

2.22

2
1
One
Two
Three
Half sentenceaccent phrase accent phrases accent phrases

Short

Sentence

NaturalLong

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202077

Discussion▪ Summary:• Develop neural ITTS based on sequence-to-sequence framework• Experimental results:- Linguistic feature of accent phrase is criticalwhen the next linguistic features are missing- The optimum incremental synthesized units was between
the three accent phrases and the half-sentence units

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

78

Approach to Problem 1&2

Incremental Machine Speech Chain

Towards Enabling Listening while Speaking
in Real-time[Sashi Novitasari, Andros Tjandra, Tomoya Yanagita, Sakriani Sakti, Satoshi Nakamura,“Incremental Machine Speech Chain Towards Enabling Listening while Speaking in Real-time,” Proc. of INTERSPEECH, Oct 2020]

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

79

Machine Speech Chain▪ Speech Chain Mechanism• ASR to TTS process using speech data only to improve TTS• TTS to ASR process using text data only to improve ASR

Full-utterance-based ASR and TTS → High delay
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

80

Incremental Machine Speech Chain▪ Objective:

Basic Framework

A closed short-term feedback loop between
incremental ASR (ISR) and incremental TTS (ITTS)• Reduce feedback delay within machine speechchain training(proposed)
ASR→ TTS
Delay

ISR→ ITTS
Delay

ASR
ISR

• Improve ISR and ITTS learning quality• Enable immediate feedback generation during
inference

Incremental Framework

goodmorning TTS

ISR

morninggoo
ITTS
d ITTS

TTS→ ASR
Delay

ITTS→ ISR
Delay

good
morning TTS

goo morning
ITTS
d ITTS

ASR

goodmorning
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

ISR

good

ISR

morning81

Learning in Incremental Machine Speech Chain▪ ISR and ITTS Joint Training• A short-term feedback loop between the components• Segment-based output passing• Unrolled processesa.

ISR-to-ITTS
For each step n, ISR predicts 𝑌෠𝑛 from 𝑋𝑛 , andthen ITTS predicts 𝑋෠𝑛 from ISR output 𝑌෠𝑛

Step n = 2

Step n = 1

𝐿𝑜𝑠𝑠𝑇𝑇𝑆𝑛=1 (𝑥𝑛=1 , 𝑥ො𝑛=1 ) 𝐿𝑜𝑠𝑠𝑇𝑇𝑆𝑛=2 (𝑥𝑛=2 , 𝑥ො𝑛=2 )𝑥ො𝑛=2 =

𝑥ො𝑛=1 =
ITTS

ITTS

𝑦ො𝑛=1 = “ a b c ”

𝑦ො𝑛=2 = “ d e ”

ISR

ISR

𝑥𝑛=1 =

𝑥𝑛=2 =

Full speech =(𝑋)
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202082

Learning in Incremental Machine Speech Chain
Step n = 1▪ ISR and ITTS Joint Training
𝐿𝑜𝑠𝑠𝐴𝑆𝑅𝑛=1 (𝑦𝑛=1 , 𝑦ො𝑛=1 )• A short-term feedback loop between the components• Segment-based output passing𝑦ො𝑛=1 = “ a b c ”• Unrolled processesa.

b.

ISR-to-ITTS
For each step n, ISR predicts 𝑌෠𝑛 from 𝑋𝑛 , andthen ITTS predicts 𝑋෠𝑛 from ISR output 𝑌෠𝑛
ITTS-to-ISR
For each step n, ITTS predicts 𝑋෠𝑛 from 𝑌𝑛 , and then
ISR predicts 𝑌෠𝑛 from ITTS output 𝑋෠𝑛

Step n = 2

𝐿𝑜𝑠𝑠𝐴𝑆𝑅𝑛=2 (𝑦𝑛=2 , 𝑦ො𝑛=2 )𝑦ො𝑛=2 =“ d e ”

ISR

ISR

𝑥ො𝑛=2 =

𝑥ො𝑛=1 =
ITTS

ITTS

𝑦𝑛=1 = “ a b c ”

𝑦𝑛=2 = “ d e ”

Full text = a b c d e f(𝑌)Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202083

Incremental Machine Speech Chain▪ ASR and TTS Results:
ASR (CER%)

TTS (L2-norm)2

Std.
(delay: 7.88 sec)

Incr.
(delay: 0.84 sec)

Std.
(delay: 103 chars)

Incr.
(delay: 30 chars)

nat-spsyn-spnat-spsyn-spnat-txt

rec-txt

nat-txt

rec-txt

Indep-trn SI-84

17.33

27.03

17.81

44.54

0.99

1.02

1.04

3.62

Indep-trn SI-284

7.16

9.60

7.97

19.99

0.75

0.77

0.84

1.31

Indep-trn (SI-84) + chain-trngreedy (SI-200)

11.21

11.52

14.23

32.43

0.80

0.82

0.86

1.35

Indep-trn (SI-84) + chain-trnteachforce (SI-200)

7.27

6.30

9.43

12.78

0.77

0.80

0.79

1.26

Data

Independent Training

Machine Speech Chain

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202084

Conclusions and Future Directions

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202085

Conclusions and Future Directions▪ Conclusions:• We have constructed a machine speech chain that can listen, speak,and listen while speaking• Currently, we mostly utilize it to achieve semi-supervised learning• On the other hand, we have also constructed ISR and ITTS
• Combined ISR and ITTS within incremental machine speech chain framework▪ Future Directions:
Develop a Real-Time Neural Machine Speech Interpreter that
Listen, Translate, Speak, andListen while Speaking and Translating
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 2020

86

Citations
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪

▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪

[Sakoe et al., 1971] – H. Sakoe and S. Chiba, “A dynamic programming approach to continuous speech recognition,” in Proc. 7th ICA, 1971
[Baum et al., 1966] – L.E Baum and T. Petrie, "Statistical Inference for Probabilistic Functions of Finite State Markov Chains," The Annals of Mathematical Statistics, 1966
[Waibel et al. 1989] – A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, “Phoneme recognition using time-delay neural networks,” IEEE Trans. on Acoustic Speech, and Signal Processing, 1989
[Hochreiter et al., 1997] – S. Hochreiter; J. Schmidhuber, "Long short-term memory," Neural Computation, 1997
[Mohri et al., 2002] – M. Mohri, F. Pereira, and M.l Riley, “Weighted finite-state transducers in speech recognition,” Computer Speech and Language, 2002
[Bourlard et al., 1993] – H. Bourlard, N. Morgan, “Connectionist Speech Recognition: A Hybrid Approach,” Kluwer Academic Publishers, 1993
[Graves et al., 2006] – A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber, ´”Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with RNN,” in Proc. ICML, 2006
[Chan et al., 2016] – W. Chan, N. Jaitly, Q. Le, and O. Vinyals, et al., “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Proc. ICASSP, 2016
[Xiong et al., 2017] -- W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu and G. Zweig, "Achieving Human Parity in Conversational Speech Recognition“, Microsoft Research Technical
Report MSR-TR-2016-71, 2017
[Saon et al., 2017] -- G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L. Lim, B. Roomi, P. Hall, "English Conversational Telephone Speech
Recognition by Humans and Machines“, ASRU 2017
[Zen et al., 2009] -- H. Zen, K. Tokuda, and A. Black, “Statistical parametric speech synthesis,” Speech Communication, 2009
[Wang et al., 2017] -- Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio et al., “Tacotron: A fully end-to-end text-to-speech synthesis model,” arXiv
preprint, arXiv:1703.10135, 2017
[Oord et al., 2016] A. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” arXiv preprint
arXiv:1609.03499, 2016
[Denes & Pinson, 1993] -- P. Denes and E. Pinson, “The Speech Chain”, ser. Anchor books. Worth Publishers, 1993
[Kikui et al. , 2003] – G. Kikui, E. Sumita, T. Takezawa, and S. Yamamoto, “Creating Corpora for Speech-to-Speech Translation,” Proc. of European Conf. on Speech Communication and Technology, 2003
[Li et al. , 2017] – C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y. Cao, A. Kannan, and Z. Zhu, “Deep speaker: an end-to-end neural speaker embedding system,” arXiv preprint arXiv:1705.02304, 2017
[Paul et al. , 1992] – D. B. Paul and J. M. Baker, “The design for the Wall Street Journal-based CSR corpus,” in Proc. of the workshop on Speech and Natural Language, ACL, 1992
[Bahdanau et al. , 2016] – D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Proc. ICASSP, 2016
[Tjandra et al. , 2017] – A. Tjandra, S. Sakti, and S. Nakamura, “Local monotonic attention mechanism for end-to-end speech and language processing,” in Proc. IJCNLP, 2017
[Hwang et al. , 2016] – K. Hwang and W. Sung, “Character-level incremental speech recognition with recurrent neural networks,” in Proc. ICASSP, 2016
[Jaitly et al. , 2016] – N. Jaitly, Q. V. Le, O. Vinyals, I. Sutskever, D. Sussillo, and S. Bengio, “An online sequence-to-sequence model using partial conditioning,” in Proc. NIPS, 2016
[Ito et al. , 2017] – K. Ito, “The LJ speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017
[Kurematsu et al. , 1990] – A. Kurematsu, K. Takeda, Y. Sagisaka, S. Katagiri, H. Kuwabara, K. Shikano, ”ATR Ja Speech Database as a Tool of Speech Recog. & Synthesis,” Speech Communication, 1990
[Sonobe et al. , 2017] – 23] R. Sonobe, S. Takamichi, and H. Saruwatari, “Jsut corpus: free large-scale japanese speech corpus for end-to-end speech synthesis,” arXiv preprint arXiv:1711.00354, 2017
Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202087

Publications
General Machine Speech Chain Framework▪ A. Tjandra, S. Sakti, S. Nakamura, "Listening while Speaking: Speech Chain by Deep Learning", in Proc. IEEE Automatic Speech Recognition and Understanding (ASRU) Workshop, 2017
▪ A. Tjandra, S. Sakti, S. Nakamura, "Machine Speech Chain with One-shot Speaker Adaptation", in Proc. INTERSPEECH, 2018
▪ A. Tjandra, S. Sakti, S. Nakamura, "End-to-end Feedback Loss in Speech Chain Framework via Straight-through Estimator", in Proc. IEEE ICASSP, 2019
▪ A. Tjandra, S. Sakti, S. Nakamura, "Machine Speech Chain," IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), Vol. 28, pp. 976-989, 2020
Multilingual Machine Speech Chain▪ S. Nakayama, A. Tjandra, S. Sakti, S. Nakamura, "Speech Chain for Semi-Supervised Learning of Japanese-English Code-Switching ASR and TTS", in Proc. SLT, 2018
▪ S. Nakayama, A. Tjandra, S. Sakti, S. Nakamura, "Zero-shot Code-switching ASR and TTS with Multilingual Machine Speech Chain,“ in Proc. IEEE Automatic Speech Recognition and Understanding (ASRU)
Workshop, 2019
▪ S. Novitasari, A. Tjandra, S. Sakti, S. Nakamura, "Cross-Lingual Machine Speech Chain for Javanese, Sundanese, Balinese, and Bataks Speech Recognition and Synthesis", in Proc. SLTU, 2020
Multimodal Machine Speech Chain▪ J. Effendi, A. Tjandra, S. Sakti, S. Nakamura, “Listening while Speaking and Visualizing: Improving ASR through Multimodal Chain,” in Proc. IEEE Automatic Speech Recognition and Understanding (ASRU)
Workshop, 2019
▪ J. Effendi, A. Tjandra, S. Sakti, S. Nakamura, "Augmenting Images for ASR and TTS through Single-loop and Dual-loop Multimodal Chain Framework," in Proc. of INTERSPEECH, pp. to appear, 2020
Incremental (Real-time) ASR and TTS▪ S. Novitasari, A. Tjandra, S. Sakti, and S. Nakamura, "Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition," in Proc. INTERSPEECH, 2019
▪ T. Yanagita, S. Sakti, S. Nakamura, "Incremental TTS for Japanese Language," in Proc. INTERSPPECH, 2018
▪ T. Yanagita, S. Sakti, S. Nakamura, "Neural iTTS: Toward Synthesizing Speech in Real-time with End-to-end Neural Text-to-Speech Framework," in Proc. SSW, 2019
Incremental (Real-time) Machine Speech Chain▪ S. Novitasari, A. Tjandra, T. Yanagita, S. Sakti, S. Nakamura, "Incremental Machine Speech Chain for Enabling Listening while Speaking in Real-time," in Proc. of INTERSPEECH, pp. to appear, 2020

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202088

Thank you

Sakriani Sakti @ AHC Labs, NAIST, Japan | Speaker Odyssey – Tutorial | November 1st-5th, 202089