1

Towards More Human-like Machine
Speech Chain
Satoshi Nakamura1,2, with
Sashi Novitasari1, and Sakriani Sakti1,21Nara Institute of Science and Technology, Japan
2RIKEN, Advanced Intelligence Project AIP, Japan

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

2

Speech Chain on Human Speech Communication
In human speech production and hearing→ Closed-loop speech chain mechanism with auditory feedback“Goodafternoon”

Sensory
nervesAuditory feedback[Denes & Pinson, 1993]

Listening

Speaking
Motornerves

Linguistic
Level
無限の可能性、ここが最先端 －Outgrow your limits－

Physiological
Level
Jan. 25. 2021

Acoustic
Level

PhysiologicalLevel

Sensory
nerves

Linguistic
Level

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

3

Delayed Auditory Feedback*1,2

DAF for stutter:– DAF device that enables a user to speak into a microphone and then hear own voicein headphones a fraction of a second later– Stuttering was corrected or bypassed while speaking under DAF.

Effects in normal speakers– DAF is used to see the structure of the auditory and verbal pathways in thebrain.
– Reduction in speaking rate, increase in intensity, and increase in fundamental
frequency in order to overcome the effects of the feedback.
– Repetition of syllables, mispronunciations, omissions, and omitted word
endings.
*1Bernard S. Lee, “Delayed Speech Feedback”, The Journal of the Acoustical Society of America 22, 824 (1950);*2 Wikipedia “Delayed Auditory Feedback”
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Topics

4

Machine Speech Chain– ASR and TTS research– ASR & TTS semi-supervised joint learning

Neural Incremental ASR and TTS– Neural Incremental ASR– Neural Incremental TTS
Incremental Speech Chain– Incremental Learning of Speech Chain

Summary
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

5Machine Speech Chain▪ Proposed Method→ Develop a closed-loop speech chain model based on deep learning“Good
afternoon”

Use the closed-loop
for ASR and TTS
Sensory
nervesAuditory feedback“How are you?”

Speaking
MotornervesAuditory feedback
Speaking

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Machine Speech Chain→ In training stage:

6

ASR and TTS teach each other using unpaired data and generate useful feedback→ In Inference stage: Possible to use ASR & TTS module independently, or dependently→ Semi-supervised learning: Allow to train with labeled and unlabeled data
→ A closed-loop architecture in which the domain of source & target are different
Two agents: (1) ASR: Speech-to-text vs (2) TTS: Text-to-speech(Dual learning NMT: both agents text2text, CycleGAN: both agents image2image)
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

7

Motivation Background
Despite the close relationship between speech perception & production → ASR and TTS

researches have progressed independently

Property

ASR

TTS

Speech features

MFCC
Mel-fbank

MGC
log F0, Voice/Unvoice, BAP

Text features

PhonemeCharacterPhoneme + POS + LEX + …
(Full context label)

Model

GMM-HMM
Hybrid DNN/HMM
End-to-end ASR

GMM-HSMM
DNN-HSMM
End-to-end TTS

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

8Machine Speech Chain
Andros Tjandra, Sakriani Sakti, Satoshi Nakamura,“Listening while Speaking: Speech Chain by Deep
Learning”,
IEEE ASRU 2017

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan8
http://www.naist.jp/

Machine Speech Chain

9

Definition:– 𝑥 = original speech, 𝑦 = original text
– 𝑥ො = predicted speech, 𝑦ො = predicted text
– 𝐴𝑆𝑅(𝑥): 𝑥 → 𝑦ො (seq2seq model transforms speech to text)– 𝑇𝑇𝑆 𝑦 : 𝑦 → 𝑥ො (seq2seq model transforms text to speech)
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

10Machine Speech Chain
Case #1: Supervised Learning with Speech-Text Data
Given a pair speech-text 𝒙, 𝒚
– Train ASR and TTS in supervised learning– Directly optimized:→ 𝐴𝑆𝑅 by minimize ℒ𝐴𝑆𝑅 𝑦, 𝑦ො→ 𝑇𝑇𝑆 by minimizing loss between ℒ 𝑇𝑇𝑆 𝑥, 𝑥ො
– Update both ASR and TTS independently

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

11Machine Speech Chain
Case #2: Semi-supervised Learning with Speech Only
– Given the unlabeled speech features 𝒙
1.
2.
3.

ℒ 𝑇𝑇𝑆 (𝑥, 𝑥)ොASR predicts the most possible
transcription 𝑦ොBased on 𝑦,ො TTS tries to reconstructspeech features 𝑥ො
Calculate ℒ 𝑇𝑇𝑆 (𝑥, 𝑥)ො between original
speech features 𝑥 and the predicted 𝑥ොPossible to improve TTS with speech onlyby the support of ASR

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

12Machine Speech Chain
Case #3: Semi-supervised Learning with Text Only
– Given the unlabeled text features 𝒚
1.
2.
3.

ℒ𝐴𝑆𝑅 (𝑦, 𝑦)ොTTS generates speech features 𝑥ො
Based on 𝑥,
ො ASR tries to reconstructtext features 𝑦ොCalculate ℒ𝐴𝑆𝑅 (𝑦, 𝑦)ො between originaltext features 𝑦 and the predicted 𝑦ොPossible to improve ASR with text onlyby the support of TTS

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Sequence-to-Sequence ASR

13

Input & output• 𝒙 = [𝑥1 , … , 𝑥𝑆 ] (speech feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑡𝑑 = decoder state at time 𝑡• 𝑎𝑡 = attention probability at time t• 𝑎𝑡 𝑠 = 𝐴𝑙𝑖𝑔𝑛 ℎ𝑠𝑒 , ℎ𝑡𝑑••

𝑎𝑡 𝑠 =

exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠𝑒 ,ℎ𝑡𝑑𝑒 𝑑σ𝑆
𝑠=1 exp 𝑆𝑐𝑜𝑟𝑒 ℎ𝑠 ,ℎ𝑡
𝑐𝑡 = σ𝑆𝑠=1 𝑎𝑡 𝑠 ∗ ℎ𝑠𝑒 (expected context)

Loss function
𝑇ℒ𝐴𝑆𝑅 𝑦, 𝑝𝑦

1= − ෍ ෍ 1(𝑦𝑡 = 𝑐) ∗ log 𝑝𝑦𝑡 [𝑐]𝑇
𝑡=1 𝑐∈[1..𝐶]

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Sequence-to-Sequence TTS
End of speechprediction

Fully connected

14

Input & output• 𝒙𝑹 = [𝑥1 , … , 𝑥𝑆 ] (linear spectrogram feature)• 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] (mel spectrogram feature)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)
Loss function

Fully connected

1 𝑆ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො = ෍
𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2
𝑆
𝑠=1
𝑆
1
෠
ℒ 𝑇𝑇𝑆2 𝑏, 𝑏 = − ෍𝑏𝑠 log(𝑏෠𝑠 ) + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠
𝑆𝑠=1ℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠

CBHG: Convolution Bank + Highway + bi-GRU

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

15

Model Optimization in Speech Chain
Combined loss:

𝑈
𝑃ℓ𝐴𝐿𝐿 = 𝛼 ℓ𝑃𝑇𝑇𝑆 + ℓ𝐴𝑆𝑅
+ 𝛽 (ℓ𝑈𝑇𝑇𝑆 + ℓ𝐴𝑆𝑅)

Loss from paired data

Loss from unpaired data

𝛼 and 𝛽 are hyper-parameters for scaling the gradientfrom paired and unpaired data

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

16

Experiments on Single-speaker
Dataset:– BTEC corpus (text), speech generated by Google TTS (using gTTS library)– Supervised training: 10000 utts (text & speech paired)– Unsupervised training: 40000 utts (text & speech unpaired)

Result:DataHyperparameter
gen.
𝛼
𝛽
mode

ASR
CER
(%)

Paired
(10k)

-

-

-

+Unpaired
(40k)

0.25
0.5
0.25
0.5

1
1
1
1

greedygreedybeam 5beam 5

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

TTS

Mel

Raw

Acc
(%)

10.06

7.07

9.38

97.7

5.83
5.75
5.44
5.77

6.21
6.25
6.24
6.20

8.49
8.42
8.44
8.44

98.4
98.4
98.3
98.3

Acc: End of speechprediction accuracy

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

17Machine Speech Chain
Andros Tjandra, Sakriani Sakti, Satoshi Nakamura,“Machine Speech Chain with One-shot Speaker
Adaptation”,
INTERSPEECH 2018

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan17
http://www.naist.jp/

18

Multi-Speaker Speech Chain
Adding a speaker embedding as conditional input for TTS

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Training Unpaired Data Scenario

19

Train with unpaired speech

Train with unpaired text

1. ASR predict best transcription 𝑦ො given 𝑥
2. SPKEMB generate speaker embedding 𝑧
3. TTS reconstructs 𝑥ො given 𝑦,
ො 𝑧

1. Sample a speaker embedding 𝑧ǁ from any speech
2. TTS generates speech feature 𝑥ො given 𝑦, 𝑧ǁ
3. ASR reconstruct text 𝑦ො given 𝑥ො

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

20

Tacotron + Multi-speaker Adaptation
Input & output• 𝒙𝑹 = 𝑥1 , … , 𝑥𝑆 , 𝒙𝑴 = [𝑥1 , … , 𝑥𝑆 ] (mel & linear spectrogram)• 𝒚 = [𝑦1 , … , 𝑦𝑇 ] (text)• 𝒛 (speaker embedding vector)
Model states• ℎ𝑒1..𝑆 = encoder states• ℎ𝑠𝑑 = decoder state at time 𝑡• 𝑎𝑠 = attention probability at time t• 𝑐𝑠 = σ𝑆𝑠=1 𝑎𝑠 𝑡 ∗ ℎ𝑡𝑒 (expected context)
Loss function
1 𝑆
𝑥𝑠𝑀 − 𝑥ො𝑠𝑀 2 + 𝑥𝑠𝑅 − 𝑥ො𝑠𝑅 2
Reconstruction MSE ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො = 𝑆 ෍
𝑠=1
1 𝑆
𝑏𝑠 log 𝑏෠𝑠 + 1 − 𝑏𝑠 log 1 − 𝑏෠𝑠
EOS cross entropy ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠ = − 𝑆 ෍
𝑠=1
𝑧, 𝑧Ƹ
Perceptual loss betweenℒ 𝑇𝑇𝑆3 𝑧, 𝑧Ƹ = 1 −𝑧 2 + 𝑧Ƹ 2

original & generated speechℒ 𝑇𝑇𝑆 𝑥, 𝑥,
ො 𝑏, 𝑏෠ = ℒ 𝑇𝑇𝑆1 𝑥, 𝑥ො + ℒ 𝑇𝑇𝑆2 𝑏, 𝑏෠ + ℒ 𝑇𝑇𝑆3 (𝑧, 𝑧)Ƹ

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Experiment Results on WSJ

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan21

http://www.naist.jp/

Topics

22

Machine Speech Chain– ASR and TTS research– ASR & TTS semi-supervised joint learning

Neural Incremental ASR and TTS– Neural Incremental ASR– Neural Incremental TTS
Incremental Speech Chain– Incremental Learning of Speech Chain

Summary
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

23

Neural Incremental Speech Recognition
Incremental Speech Recognition● ISR begins the speech recognition without waiting the speech to finish (low delay)o Recognize the speech part-by-part in several incremental steps
o Input: a short part of the speech● Challenge: How to do an incremental step?
1)

Input boundary decision
When the transcription of a short speech partcan be produced?
Give
output?

2)

Output boundary decision
When to stop the output prediction of the current
speech part and and move to the next?
ABCD…

SpeechWhen tostop?

SpeechGive
output?

When to
stop?

KLMN…

Need to learn short input-short output alignments
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan23
http://www.naist.jp/

24

Neural Incremental Speech Recognition
Incremental Speech Recognition

Related Works

A. Statistical approach (Pipeline)❖ Hidden Markov model (HMM) ASR [Rabiner, 1989; Juang and Rabiner, 1991]❖ 3 parts: Acoustic model, lexicon, language model

Featureextraction

Acoustic
model

Input speech
HMM

Lexicontoday: /təˈdeɪ/
going: /ˈɡəʊɪŋ/
Pronunciation
labelsLanguagemodel“Today I am going totalk about speech
recognition”

N-gram model

Output text

❖ Low delay speech recognition by performing left-to-right input processing (unidirectional)❖ Not end-to-end Young, 2008
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan24
http://www.naist.jp/

25

Neural Incremental Speech Recognition
How to achieve an ISR system that can:
1. reduce delay,
2. keep the system complexity, and3. maintain a close performanceof the standard neural ASR system?
Proposal
Neural ISR construction by employing sources (architecture, knowledge)
from standard neural ASR.

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan25
http://www.naist.jp/

26

Neural Incremental Speech Recognition

Sashi Novitasari, Andros Tjandra, Sakriani Sakti, Satoshi
Nakamura,“Sequence-to-sequence Learning via Attention Transfer
for Incremental Speech Recognition”,

Interspeech 2019
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan26
http://www.naist.jp/

27

Attention-Transfer ISR
Attention-Transfer Incremental Speech Recognition (AT-ISR)[Novitasari et al., 2019]●

AT-ISR Training

Aim

This isboundary!

ISR (student) learns to mimic the attention-based alignment generated
by a standard seq2seq ASR (teacher)oo

ISR architecture
Incremental step

: Same as the teacher (seq2seq): Learn through attention transfer from theteacher ASR● Attention transfer : Attention knowledge transfer from teacher tostudent modelo Prev. works → image recognition tasks▪ Teach another model [Zaguruyko and Komodakis, 2017]▪ Domain transfer (image to video) [Li et al., 2017]o Has not been utilized for ISR construction yet

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

ABCDEFG

ABCD|

EFG|

ASR

ISR

ISR

Att

This isboundary!

Standard ASR(TEACHER)All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, JapanISR(STUDENT)27

http://www.naist.jp/

28

Attention-Transfer ISR
Attention Transfer
Train ISR (student) to learn the attention-based alignment froma standard seq2seqASR (teacher)1) Extract speech-text alignment from attention matrix generated by
the teacher ASR during teacher-forcing text generation(alignment pair = high attention score):

2) Train the ISR by using 𝒀𝑛 + </m> as the target of 𝑿𝑛

</m>

</m>

𝒀2

𝒀1

Teacher ASR attention matrix


Alignment
Char
Token
ID

Seg. ID(n)

Speech seg.
(𝑿𝒏 )Text seg.
(𝒀𝒏 )1

x1 – xw

y1 – y32

xw+1 – xw+2

y4 – y5(etc.)

𝑿1

𝑿2
n=1

Speech Frame Block ID(1 block = W frames)
無限の可能性、ここが最先端 －Outgrow your limits－

n=2

ISR delay can be managed by changing 𝑿𝑛 and 𝒀𝑛 size during
training
e.g. higher delay : combine several segments into one
Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan28

http://www.naist.jp/

29

Attention-Transfer ISR
AT-ISR Recognition Method</m>Given: Full speech (𝑿), length 𝑆●

Recognize the speech segment-by-segment sequentially basedon a fix-sized input window
For each incremental recognition step n:

Decoder

1.

Encode 𝑿𝑛 , a 𝑊 speech frames from 𝑿 (𝑊 < 𝑆)

Attention2.

Decode for 𝒀𝑛 that aligns with 𝑿𝑛 , until an end-of-block</m> token is predicted or max. length is reached
o Attend the input 𝑿𝑛

Shift the input window 𝑊 frames by keeping the model’s
state
𝑆(Total step number: 𝑁 = )
3.

𝑊

●

●

Incremental step:● Input boundary
window● Output boundary

</m>

𝒀𝑛

●

●

Step n = 2

Step n = 1
Encoder𝑿𝑛: last speech frame in the input: </m> token in the output text

W frames

W frames

Alignment learning → Attention transfer

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan29

http://www.naist.jp/

30

AT-ISR Performance
Model Configuration•

AT-ISR/Teacher ASR structure: Seq2seq (identical)
y1

y2

yT

y3

Decoder
LSTM•

AT-ISR with input overlap :o Main frames : Aligns with output text seg.
o Look-ahead frames : Next to the main input(contextual input)

Emb.

time
y0

Hierarchicalsub-sampling
8 speechframes into 1
encoder state

y1

yT-1

y2

Attention

Encoder
BiLSTM

Step 1
input(overlap)

BiLSTM
Step 2
input

BiLSTM[main] [ahead]FNN

AT-ISR basic incremental unit
8 speech frames = 1 block (0.14 sec)

x1

無限の可能性、ここが最先端 －Outgrow your limits－

x2 x3

x4

x5

x6

x7

x8

Jan. 25. 2021

xS-1 xS

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan30

http://www.naist.jp/

AT-ISR Performance

31

Evaluation Setting
ISR performance evaluation was made by comparing various model:● Non-incremental ASR : Toplineo Standard seq2seq ASR (Our Att Enc-Dec; teacher)o Other existing neural ASR●

Incremental ASR:o Baseline neural ISR:
− Seq2seq ISR without attention transfer− Incremental steps were taught by using alignments from forced-alignment by HMM ASRo Proposed ISR: AT-ISR (attention transfer; student)o Other existing neural ISR: Unidirectional LSTM + CTC [Hwang and Sung, 2016]

Evaluation metric:o CER, WER
o Delay (speech input size)
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan31
http://www.naist.jp/

32

AT-ISR Performance
Speech recognition performance ofcharacter-level models trained on WSJ dataset

ResultDelay (sec)Model

CER (%)
Input

Computation●

Avg. utterance length: 7.88 sec●

Machine: Intel® CoreTM i7-9700K CPU @ 3.60GHz

Non-incremental ASR (Topline)
Att Enc-Dec (ours)
BiLSTM-CTC [1]Joint CTC+Att [1](NVIDIA GeForce RTX 2080Ti GPU)

7.88 (avg)0.32 (avg)

6.26
8.97
7.36

Baseline neural ISR

Input/step: 1 m + 1 laInput/step: 1 m + 4 la

0.24
0.54

0.02
0.05

20.15
11.95

0.24
0.54

0.02
0.05

18.37
7.52

Proposed AT-ISR
Input/step: 1 m + 1 laInput/step: 1 m + 4 la
Other existing neural ISR
LSTM-CTC beam search
[2]

-

-

10.96

CER
diff.:

1.3%●

ISR performance limitation: short-segmentbased recognition (incomplete information)●

Contextual input (la) improves performance
AT-ISR performs well with a short delay bylearning non-incremental ASR’s knowledge*Notem= main input blockla= look-ahead block (contextual input)
1 block = 8 frames = 0.14 sec[1] Suyoun Kim, Takaaki Hori, and ShinjiWatanabe. Joint CTC-attention based end-to-end speech recognition using multitask learning. In Proceedings of ICASSP, pages 4835-4839, New Orleans, USA, 2017.
[2] Kyuyeon Hwang and Wonyong Sung. Character-level incremental speech recognition with recurrent neural networks. In Proceedings of ICASSP, pages 5335 - 5339, Shanghai, China, 2016.

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan32

http://www.naist.jp/

33

AT-ISR Performance and Delay
ISR Delay

WER (%) of AT-ISR trained on TED-LIUM dataset

How did the ISR delay affected
the ISR performance?
Trade-off: Higher delay, lower WER●

Subword-level ISRoo●Lower WER than character-level ISR
Keep word context longer thancharacters

Character-level ISRooMaintains the teacher’s
performance better than the
subword-level ISR
ISR with delay 2.04 starts to have aclose performance to the teacher
ASR

Subword-level model

50
46.14
45.39

40

WER %

●

Character-level model

33.94
31.06

32.24

30

28.76
28.26

27.79

27.38

27.7
25.27

24

20
0.44

0.64

0.84

2.04

3.94

7.58

(25% S)(50% S)(100% S;
Topline)Delay (sec)*S = average full-utterance length (7.58 sec)33
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

34

Neural Incremental Speech Recognition
Summary – AT-ISR
Neural ISR system (AT-ISR) with a low recognition delay without increasing the complexity of thestandard ASR system
1. AT-ISR with delay < 1 sec. achieved a close performance to standard ASR with delay > 7 sec.
2. AT-ISR as an ISR framework with an efficient development mechanism and reliable
performance via attention transfer that applies an identical architecture as the standard ASR

Recent ISR Trend● Streaming ASR with RNN-Transducer (RNN-T) [Saitnah et al., 2020; Li et al., 2020]● Streaming transformer ASR [Miao et al., 2020; Moritz et al., 2020; Tsunoo et al., 2020]

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Topics

35

Machine Speech Chain– ASR and TTS research– ASR & TTS semi-supervised joint learning

Neural Incremental ASR and TTS– Neural Incremental ASR– Neural Incremental TTS
Incremental Speech Chain– Incremental Learning of Speech Chain

Summary
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Incremental Text-to-speech

36

Incremental Text-To-Speech(iTTS)
Speech is synthesizes a speech in shorter delay (e.g. word).
It can synthesize a speech before finishing text input.

Challenges
How to improve speech quality?
Speech quality of Incremental TTS
How to estimate target prosody from an incomplete sentence?
target prosody is typically calculated from long-window features. (e.g. co-articulation)-> predicts next information(e.g. word) at step of input-to-acoustic-features.
-> wait next word when synthesizing a current word.
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Related Works of iTTS(1/2)

37

Statistical approach (pipeline)
Hidden Markov model TTS[Baumann et al., 2014.],[Pouget et al., 2015],[Yanagita, et al., 2018]
No neural End-to-end iTTS approach

Text

Text
analysis

Acoustic model
Linguistic(HMM)
featureEstimation ofduration andacoustic feature

Synthesized
waveform

Vocoder(Digital filter)

previous/current/next phoneme
previous/current/next POS
previous/current/next accentual info.

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

38

Related Works of iTTS(2/2)
End-to-end TTS [Wang, et al., 2017.], [Sotelo, et al., 2017], [Shen, et al., 2018.]
Encoder-decoder with an attention mechanism-> Output prediction starts after the input sequence.
The speech is also synthesized Sentence-by-sentence.
-> It can generate High quality speech close to human.
Challenge of the neural iTTS system.
More natural synthesized speech

Neural iTTS[Yanagita et al., 2019]no wait next word for synthesis.
Control output sequence with stop flag

無限の可能性、ここが最先端 －Outgrow your limits－

Prefix-to-Prefix Framework [Ma et al., 2020]
wait next word for synthesis.
Control output sequence with attention weight and stop flag
One word look ahead at least for synthesis.

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

39

Neural iTTS

Tomoya Yanagita, Sakriani Sakti and Satoshi Nakamura,“Neural iTTS: Toward Synthesizing Speech in Real-timewith End-to-end Neural Text-to-Speech Framework”,
10th Speech Synthesis Workshop (SSW10) , Sep. 2019

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

40

End-to-End TTS
Encoder

Decoderwith attention

We use Tacotron[Wang, et al., 2017.].
Stop flag prediction to control output seq. is also used.

•
•
•
•
•
•

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

xN : Input sequence (N length)hN : hidden representatiaon of encode
cs : context vector (S length )ys : mel spectrogram
Ls : Linear spectrogram
Ss : Stop flag

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

41

Proposed method
: sentence start
: sentence end<m>: middle sentence start</m>: middle sentence end

Dataset sentences are divided into three parts.
– use location symbol to indicate locations– use all data for training

Inference: Ex. “Today we talk about TTS.”
Encoder

Encoder

Encoder

Encoder
 Today 
<m> we 

 Today </m><m> we </m>

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Result of English MOS

42

Still big gap between natural speech and synthesized speech.
Three words unit has the best En. score.

4.98

5
4
3.17

3.12

3.01

Three
words

Half
sentence

Sentence

2.79

3
2.08

2
1
One
word

Two
words

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

NaturalspeechAll Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Result of Japanese MOS

43

Still big gap between natural speech and synthesized speech.
Half sentence unit ≒ the full sentence units (Ja.).
4.95

5
4
2.81

3

2.88

3.13

3.18

2.22

2
1
One
Two
Three
Half sentenceaccent phrase accent phrases accent phrases

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

Sentence

NaturalAll Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Topics

44

Machine Speech Chain– ASR and TTS research– ASR & TTS semi-supervised joint learning

Neural Incremental ASR and TTS– Neural Incremental ASR– Neural Incremental TTS
Incremental Speech Chain– Incremental Learning of Speech Chain

Summary
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

45

Incremental Speech Chain

Sashi Novitasari, Andros Tjandra, Tomoya Yanagita,
Sakriani Sakti and Satoshi Nakamura,“Incremental Machine Speech Chain Towards Enabling
Listening while Speaking in Real-time”,

INTERSPEECH 2020
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

43

Incremental Machine Speech Chain
Closed short-term feedback loop between
incremental ASR (ISR) and incremental TTS (ITTS)• Reduce feedback delay within machine speechchain training• Improve ISR and ITTS learning quality• Enable immediate feedback generation during

Basic Framework

Incremental Framework(proposed)

ASR→ TTS
Delay

ISR→ ITTS
Delay

ASR

good
morning TTS

ISR

ISR

good

morning

ITTS

inferenceMove a step closer for ASR and TTS that can adapt toreal-time environment unsupervisedly→ Similar to human

TTS→ ASR
Delay

ITTS

ITTS→ ISR
Delay

good
morning TTS

goodmorning

ITTS

ASR

ITTS

ISR

ISR

good
good
morningmorning
Unrolled processes in machine speech chain loop
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

47

Incremental Machine Speech Chain: Training
Two training phases:
1. ISR and ITTS supervised-independent training
2. ISR and ITTS joint training via short-term feedback loop

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

45

ISR and ITTS Independent Training
Attention alignment from
non-incremental ASR• Incremental : Predict a complete output sequence in N steps, for eachstep n :1. Encode a segment of input from input window
2. Decode and predict a segment of output
3. Shift the input windows

Text
Token
ID

• ISR and ITTS training by attention transfer from non-incremental ASR[Novitasari et al., 2019] → same alignment for ISR and ITTS

ISR
Output Text
(Yn)

Step n = 2

a b c </m>d e </m>
ISR

Dec

Input Speech(Xn)x1,… x8

ITTS

Step n = 2

x1,… x8

x9,… x16

Dec

Input Text
(Yn)x9,… x16

Jan. 25. 2021

ITTS

Dec

Att

Enc

Full speech(𝑋)
無限の可能性、ここが最先端 －Outgrow your limits－

Step n = 1

Att

Att
Enc

Output Speech
(Xn)

Dec

Speech Frame Block ID
Alignment
info.

ITTS

Step n = 1

ISR

Alignment
info.

Att

Enc

Enc

<m> a b c </m><m> d e </m>Full text
abcdef(𝑌)
All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

46

Learning Approach
Exploration on 2 learning approaches:A) Semi-supervised incremental machine speech chain
1) ISR/ITTS independent training : supervised
2) ISR/ITTS joint training: unsupervised (unlabeled data)→ Same as the original basic machine speech chain

(B)

(A)

𝑥ො𝑛 =

B) Supervised incremental machine speech chain
1) ISR/ITTS independent training : supervised
2) ISR/ITTS joint training : supervised (labeled data)

ITTS

ITTS

𝑦ො𝑛 = “ a b c ”“abc”

Unsupervised
GreedyISRSupervised
Teacherforcing

ISR
Correcttext

𝑥𝑛 =Unrolled process examples in joint training(ITTS-to-ISR follows similar mechanism)
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

47

Speech Chain Result (SI-84)
ASR (CER%) and TTS (log Mel-spectrogram L2 loss) performances
TTS (L2-norm)2

ASR (CER%)
Data(delay: 7.88 sec)

Incr.
(delay: 0.84 sec)

Std.
(delay: 103 chars)

Incr.
(delay: 30 chars)

nat-spnat-spnat-txt

nat-txt

Std.
syn-spsyn-sprec-txt• Baseline:
ISR and ITTS indep-trn SI-84• Topline:
Standard systems (std.) indep-trn SI-284• Input type:rec-txt

nat-sp

ISR

Independent Training
Indep-trn SI-84

17.33

27.03

17.81

44.54

0.99

1.02

1.04

3.62

Indep-trn SI-284

7.16

9.60

7.97

19.99

0.75

0.77

0.84

1.31

rec-txt“abc”

Machine Speech Chain
Indep-trn (SI-84) + chaintrn-greedy (SI-200)

11.21

11.52

14.23

32.43

0.80

0.82

0.86

1.35

Indep-trn (SI-84) + chaintrn-teachforce (SI-200)

7.27

6.30

9.43

12.78

0.77

0.80

0.79

1.26

ITTS
nat-txt

ITTS“abc ”

ISR“abc”

Incremental machine speech chain•

Incremental system able to reduce delay with a close performance to nonincremental system•

Incremental machine speech chain improves ISR and ITTS

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

synsp

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

51

Incremental Machine Speech Chain

Incremental machine speech chain
Short-term feedback loop for ISR/ITTS development by mimicking human speech chaino Reduced the delay with a close performance to the basic frameworko Improve ISR and ITTS (natural/synthetic input)
o Synthetic input processing: demonstration of real-time feedback generation
A step to achieve system that can listen while speaking in real-time

無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

52

Summary
Machine Speech Chain– ASR and TTS research– ASR & TTS semi-supervised joint learning

Neural Incremental ASR and TTS– Neural Incremental ASR– Neural Incremental TTS
Incremental Speech Chain– Incremental Learning of Speech Chain

Future work
– Semi-supervised learning and online incremental learning– Much lower delay– Application to Incremental Dialogue System and Speech Translation System– Multi-modality, Code Switching
無限の可能性、ここが最先端 －Outgrow your limits－

Jan. 25. 2021

All Rights Reserved ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/