INTERSPEECH 2020

Incremental Machine Speech Chain
Towards Enabling Listening while Speaking
in Real-time

Copyright © 2020 NAIST Japan, All rights reservedOutline

Copyright © 2020 NAIST Japan, All rights reserved

I.

Introduction

II.

Incremental Machine Speech Chain

III.

Experiments

IV.

Conclusion

INTERSPEECH 20202

I.

Introduction

II. Incremental Machine
Speech ChainI.

Introduction

III. Experiments
IV. Conclusion

Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020

3Background

ASR and TTS• Spoken language technologies:

ASR

o Automatic speech recognition (ASR)“Hello”

o Text-to-speech synthesis (TTS)• Crucial for human-machine interaction• Remarkable performance requires a lot of speech-text paired data

“Hello”

TTS

ASR and TTS systems
Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020

4

Background

Machine Speech Chain[Tjandraet al., 2017]• Semi-supervised ASR and TTS training via closed
feedback loop

• ASR/TTS : standard attention-based seq2seq network• 2 training phases:
1) ASR/TTS supervised independent training
2) ASR/TTS unsupervised joint training withfeedback loopunroll

ASR→ TTS (speech only)

TTS→ ASR (text only)

timetime

𝐿𝑜𝑠𝑠𝐴𝑆𝑅

𝐿𝑜𝑠𝑠𝑇𝑇𝑆• Full-utterance-based ASR and TTS  High delayTTSASRTTSASR

5Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020

Background

Human Speech Chain
Human speech chain [Denes, 1993]• Feedback loop between speech production and hearing
systems• Real-time process  immediate adaptation• Feedback delay causes a disturbance during speaking
Challenge in mimicking human speech chain for machine
Speech generation or recognition and feedback generation based on incompletesequence information with minimum delay

Propose : Incremental Machine Speech Chain

6Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020

I.

Introduction

II. Incremental Machine
Speech Chain

II.

Incremental Machine Speech Chain

III. Experiments
IV. Conclusion

Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 20207

Objective

ProposeIncremental Machine Speech Chain
Basic Framework

Incremental Framework(proposed)

Closed short-term feedback loop between
incremental ASR (ISR) and incremental TTS (ITTS)

ASR TTS
Delay• Reduce feedback delay within machine speech

ISR ITTS
Delay

ASR
ISRchain training

TTS• Improve ISR and ITTS learning quality• Enable immediate feedback generation during

TTS ASR
DelayinferenceMove a step closer for ASR and TTS that can adapt toreal-time environment unsupervisedly Similar to human

ISR

ITTS

ITTS

ITTS ISR
Delay

ITTS

TTS

ASR

ITTS

ISR

ISR

Unrolled processes in machine speech chain loop
Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 20208

Incremental Machine Speech ChainComponents

ISR
Delay

Incremental ASR (ISR): Low delay ASR••

Hidden Markov model ASR
End-to-end ISR with attention-based seq2seq modelo Neural transducer [Jaitly et al, 2016]o Attention-transfer ISR [Novitasari et al., 2019]ISRISR

good morning

Incremental (ITTS): Low delay TTS••...

Hidden Markov model TTS
End-to-end ITTS with attention-based seq2seq modelo Neural ITTS [Yanagita et al., 2019]o ITTS based on prefix-to-prefix framework [Ma et al., 2019]• Performance limitation due to short-input-based processing• Previous: independent development

ITTS

goodmorning

ITTS

ITTS

...

9Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020

Incremental Machine Speech Chain

Training Mechanism
2 training phases:
1. ISR and ITTS supervised-independent training
2. ISR and ITTS joint training via short-term feedback loop

10Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020

Incremental Machine Speech Chain Training

1. ISR and ITTS Independent Training

Attention alignment fromstandard ASR• Incremental : Predict a complete output sequence in N steps.
For each step n :1. Encode a segment of input from input window
2. Decode and predict a segment of output
3. Shift the input windows

TextTokenID• ISR and ITTS training by attention transfer from standard non-incremental ASR [Novitasari et al., 2019]  same alignment for ISR and ITTS

ISR
Output Text(Yn)

Step n = 2

a b c </m>d e </m>

ISR

ISR

Dec

x1,… x8x9,… x16

Copyright © 2020 NAIST Japan, All rights reserved

Step n = 2

x1,… x8x9,… x16

Dec

ITTS

Dec

Att

Att
Enc

Alignment
info.

Step n = 1

ITTS

Enc

Full speech(𝑋)

Output Speech
(Xn)

Dec

Att

Input Speech(Xn)

ITTS

Step n = 1

Speech Frame Block ID

Alignment
info.

Input Text(Yn)Full text
(𝑌)

Enc

Enc

<m> a b c </m><m> d e </m>abcdef

INTERSPEECH 2020Att11

Incremental Machine Speech Chain Training

2. ISR and ITTS Joint Training

Step n = 1

𝐿𝑜𝑠𝑠𝑇𝑇𝑆𝑛=1 (𝑥𝑛=1 , 𝑥𝑛=1 ) 𝐿𝑜𝑠𝑠𝑇𝑇𝑆𝑛=2 (𝑥𝑛=2 , 𝑥𝑛=2 )• Short-term feedback loop between the components• Segment-based output passing• Unrolled processes
a.
a.

b.

ISR-to-ITTS
ISR-to-ITTS
For each step n, ISR predicts 𝑌𝑛 from 𝑋𝑛 , andthen ITTS predicts 𝑋𝑛 from ISR output 𝑌𝑛

𝑥𝑛=2 =𝑥𝑛=1 =ITTSITTS

𝑦𝑛=1 = “ a b c ”

𝑦𝑛=2 = “ d e ”

ISR

ISR

ITTS-to-ISR

𝑥𝑛=1 =
Full speech =(𝑋)Copyright © 2020 NAIST Japan, All rights reserved

Step n = 2

INTERSPEECH 2020

𝑥𝑛=2 =12

Incremental Machine Speech Chain Training

2. ISR and ITTS Joint Training• Short-term feedback loop between the components• Segment-based output passing• Unrolled processesa.

ISR-to-ITTS
For each step n, ISR predicts 𝑌𝑛 from 𝑋𝑛 , andthen ITTS predicts 𝑋𝑛 from ISR output 𝑌𝑛

b. ITTS-to-ISRb.
ITTS-to-ISR
For each step n, ITTS predicts 𝑋𝑛 from 𝑌𝑛 , andthen ISR predicts 𝑌𝑛 from ITTS output 𝑋𝑛

Step n = 1

Step n = 2

𝐿𝑜𝑠𝑠𝐴𝑆𝑅𝑛=1 (𝑦𝑛=1 , 𝑦𝑛=1 )𝐿𝑜𝑠𝑠𝐴𝑆𝑅𝑛=2 (𝑦𝑛=2 , 𝑦𝑛=2 )𝑦𝑛=1 = “ a b c ”

𝑦𝑛=2 =“ d e ”

ISR

𝑥𝑛=2 =

𝑥𝑛=1 =
ITTS

ITTS

𝑦𝑛=1 = “ a b c ”

𝑦𝑛=2 = “ d e ”

Full text = a b c d e f(𝑌)Copyright © 2020 NAIST Japan, All rights reserved

ISR

INTERSPEECH 202013

Incremental Machine Speech Chain

Learning Approach(B)

(A)

𝑥𝑛 =

Exploration on 2 learning approaches:A) Semi-supervised incremental machine speech chain
1) ISR/ITTS independent training : supervised
2) ISR/ITTS joint training: unsupervised (unlabeled data)B) Supervised incremental machine speech chain
1) ISR/ITTS independent training : supervised
2) ISR/ITTS joint training : supervised (labeled data)

ITTS

ITTS

𝑦𝑛 = “ a b c ”“abc”

Unsupervised
GreedyISRSupervised
Teacherforcing

ISR
Correcttext

𝑥𝑛 =Unrolled process examples in joint training(ITTS-to-ISR follows similar mechanism)
Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 202014I.

Introduction

II. Incremental Machine
Speech Chain

III. Experiments

III. Experiments
IV. Conclusion

Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 202015

Experiments

DatasetWall Street Journal CSR Corpus [Paul and Baker, 1992]• Language : English
Training sets:ooo

SI-84
SI-200SI-284: 16 hours of speech, 83 speakers: 66 hours of speech, 200 speakers: si84 + si200

Dev. set: dev93

Eval. set

: eval92• Character-level• Speech features: 80-dims log Mel spectrogram (window: 50 msec, shift: 12.5 msec)
16
Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020

Experiments

Model Configuration* Same architecture for standard (non-incremental) and incremental models

ASR
y1

Transcription

y2y3

TTS

yT

Tacotron 2 [Wang et al., 2017] structure with speaker embedding [Tjandra et al., 2018]

Decoder
CBHG

LSTM

Speech features

CharEmb.

x1,…, x4x5,…, x8

xT-3, …, xT

Decodery0

y1y2Linear Proj.

yT-1

2 LSTM

Attention
Encoder
Hierarchicalsub-sampling
8 featureframes into 1
encoderstate

Mel-to-linear

BiLSTM

x0

BiLSTMInput/step• ISR: 0.84 sec

BiLSTM• Std. ASR : full-utterance(avg. 7.88 sec)

FNNSpeech features

Speaker Emb.

2 Pre-Netx1,…, x4
BiLSTM

Input/step• ITTS : avg. 30 chars

3 Conv• Std. TTS : full-sentence(avg. 103 chars)

Transcription

INTERSPEECH 2020

Attention

Encoder

Char Emb.

x1 x2 x3 x4 x5 x6 x7 x8 xS-1 xS

Copyright © 2020 NAIST Japan, All rights reserved

xT-7,…, xT-4

y0y1 y2yT-1

17

Result
ASR (CER%) and TTS (log Mel-spectrogram L2 loss) performances
TTS (L2-norm)2

ASR (CER%)(delay: 7.88 sec)

Incremental(delay: 0.84 sec)

Standard(delay: 103 chars)

Incremental(delay: 30 chars)

nat-spsyn-spnat-spsyn-spnat-txt

rec-txt

nat-txt

rec-txt

Indep-trn SI-84

17.33

27.03

17.81

44.54

0.99

1.02

1.04

3.62

Indep-trn SI-284

7.16

9.60

7.97

19.99

0.75

0.77

0.84

1.31

Data

Standard

Independent Training• Baseline
ISR and ITTS indep-trn SI-84• Topline
Standard systems indep-trn SI-284• Proposed
Incremental machine speech chain• Input type:nat-sp

ISR

Machine Speech Chain
rec-txt

Indep-trn (SI-84) +chain-trn-greedy (SI-200)

11.21

Indep-trn (SI-84) +chain-trn-teachforce(SI-200)

7.27

•

11.52

14.23

32.43

0.80

0.82

0.86

1.35

“”
ITTS

6.30

9.43

12.78

0.77

0.80

Incremental machine speech chain

1.26
nat-txt

ITTS

“”

ISR

“”

syn-spo Improved ISR and ITTS
o Shorter delay with a close performance to the standard system
Copyright © 2020 NAIST Japan, All rights reserved

0.79

INTERSPEECH 2020

18

I.

Introduction

II. Incremental Machine
Speech Chain

IV. Conclusion

III. Experiments
IV. Conclusion

Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 202019

Conclusion
Incremental machine speech chain
Short-term feedback loop for ISR/ITTS development by mimicking human speech chaino Reduced the delay with a close performance to the basic frameworko Improve ISR and ITTS (natural/synthetic input)
o Synthetic input processing: demonstration of real-time feedback generation

20
Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020

Thank you

21
Copyright © 2020 NAIST Japan, All rights reserved

INTERSPEECH 2020