Real-time Neural Machine Speech Chain

Copyright © 2021 NAIST Japan, All rights reservedOutline

Copyright © 2021 NAIST Japan, All rights reserved

I.

Introduction

II.

Incremental Machine Speech Chain

III.

Experiments

IV.

Conclusion

2I.

Introduction

II. Incremental Machine
Speech ChainI.

Introduction

III. Experiments
IV. Conclusion

Copyright © 2021 NAIST Japan, All rights reserved3

Background

ASR and TTS

Overview of Machine Speech Chain• ASR and TTS closely related to each other Research trends : independent development

Machine Speech Chain [Tjandra et al., 2017]• Semi-supervised ASR and TTS training via closed
feedback loop

• Inspired from human speech chain [Denes, 1993]o Listening while speaking

Copyright © 2021 NAIST Japan, All rights reserved

4

Background

ASR and TTS
Machine speech chain•

2 training phases:
1) ASR/TTS supervised independent training
2) ASR/TTS unsupervised joint training with feedback loop
- 2 unrolled processes inside the feedback loop:A)

ASR-to-TTS (speech only)B)

TTS-to- ASR (text only)
timetime

𝐿𝑜𝑠𝑠𝐴𝑆𝑅

𝐿𝑜𝑠𝑠𝑇𝑇𝑆
TTS

ASR

TTS

Copyright © 2021 NAIST Japan, All rights reserved

ASRCurrent framework is for the fullutterance-based ASR/TTS
 High delay

5Background

Human Speech Chain
Human speech chain [Denes, 1993]• Feedback loop between speech production and hearingsystems• Real-time process  immediate adaptation• Feedback delay causes a disturbance during speaking
Challenge in mimicking human speech chain for machine
Speech generation or recognition and feedback generation based on incompletesequence information with minimum delay

Propose : Incremental Machine Speech Chain

Copyright © 2021 NAIST Japan, All rights reserved6

I.

Introduction

II. Incremental Machine
Speech Chain

II.

Incremental Machine Speech Chain

III. Experiments
IV. Conclusion

Copyright © 2021 NAIST Japan, All rights reserved

7

Objective

ProposeIncremental Machine Speech Chain
Basic Framework

Closed short-term feedback loop between
incremental ASR (ISR) and incremental TTS (ITTS)

Incremental Framework(proposed)

ASR TTS
Delay

ISR ITTS
Delay• Reduce feedback delay within machine speechchain training

ASR
ISR

• Improve ISR and ITTS learning quality• Enable immediate feedback generation during
inferenceMove a step closer for ASR and TTS that can adapt toreal-time environment unsupervisedly Similar to human

TTSTTS ASRDelay

ISR

ITTS

ITTS

ITTS ISRDelayITTS

TTSASR

ITTS

ISR

ISR

Unrolled processes in machine speech chain loop
Copyright © 2021 NAIST Japan, All rights reserved

8Incremental Machine Speech Chain

Components

ISR
Delay

Incremental ASR (ISR): Low delay ASR••

Hidden Markov model ASR
End-to-end ISR with attention-based seq2seq modelo Neural transducer [Jaitly et al, 2016]o Attention-transfer ISR [Novitasari et al., 2019]

Incremental (ITTS): Low delay TTS••Hidden Markov model TTS
End-to-end ITTS with attention-based seq2seq modelo Neural ITTS [Yanagita et al., 2019]o ITTS based on prefix-to-prefix framework [Ma et al., 2019]• Performance limitation due to short-input-based processing• Previous: independent development

Copyright © 2021 NAIST Japan, All rights reserved

...
ISR

ISR

good morning
ITTS

goodmorning

ITTS

ITTS

...

9

Incremental Machine Speech Chain

Training Mechanism
2 training phases:
1. ISR and ITTS supervised-independent training
2. ISR and ITTS joint training via short-term feedback loop

Copyright © 2021 NAIST Japan, All rights reserved

10Incremental Machine Speech Chain Training

1. ISR and ITTS Independent Training

Attention alignment fromstandard ASR• Incremental : Predict a complete output sequence in N steps.
For each step n :1. Encode a segment of input from input window
2. Decode and predict a segment of output
3. Shift the input windows

TextTokenID• ISR and ITTS training by attention transfer from standard non-incremental ASR [Novitasari et al., 2019]  same alignment for ISR and ITTS

ISROutput Text(Yn)

Step n = 2a b c </m>d e </m>

ISR

ISR

Dec

x1,… x8x9,… x16

Copyright © 2021 NAIST Japan, All rights reserved

Step n = 2

x1,… x8x9,… x16

Dec

ITTS

Dec

Att

Att
Enc

Alignment
info.

Step n = 1

ITTS

Enc

Full speech(𝑋)

Output Speech
(Xn)

Dec

Att

Input Speech
(Xn)

ITTS

Step n = 1

Speech Frame Block ID

Alignment
info.

Input Text
(Yn)
Full text
(𝑌)

Att

Enc

Enc

<m> a b c </m><m> d e </m>abcdef11

Incremental Machine Speech Chain Training

2. ISR and ITTS Joint Training• Short-term feedback loop between the components• Segment-based output passing• Unrolled processesa.
a.

b.

ISR-to-ITTS
ISR-to-ITTS
For each step n, ISR predicts 𝑌𝑛 from 𝑋𝑛 , andthen ITTS predicts 𝑋𝑛 from ISR output 𝑌𝑛

Step n = 1

𝐿𝑜𝑠𝑠𝑇𝑇𝑆𝑛=1 (𝑥𝑛=1 , 𝑥𝑛=1 ) 𝐿𝑜𝑠𝑠𝑇𝑇𝑆𝑛=2 (𝑥𝑛=2 , 𝑥𝑛=2 )

𝑥𝑛=2 =𝑥𝑛=1 =ITTS

ITTS

𝑦𝑛=1 = “ a b c ”

𝑦𝑛=2 = “ d e ”

ISR

ISR

ITTS-to-ISR

𝑥𝑛=1 =
Full speech =(𝑋)Copyright © 2021 NAIST Japan, All rights reserved

Step n = 2

𝑥𝑛=2 =12Incremental Machine Speech Chain Training

2. ISR and ITTS Joint Training• Short-term feedback loop between the components• Segment-based output passing• Unrolled processesa.

ISR-to-ITTS
For each step n, ISR predicts 𝑌𝑛 from 𝑋𝑛 , andthen ITTS predicts 𝑋𝑛 from ISR output 𝑌𝑛

b. ITTS-to-ISRb.
ITTS-to-ISR
For each step n, ITTS predicts 𝑋𝑛 from 𝑌𝑛 , andthen ISR predicts 𝑌𝑛 from ITTS output 𝑋𝑛

Step n = 1

Step n = 2

𝐿𝑜𝑠𝑠𝐴𝑆𝑅𝑛=1 (𝑦𝑛=1 , 𝑦𝑛=1 )𝐿𝑜𝑠𝑠𝐴𝑆𝑅𝑛=2 (𝑦𝑛=2 , 𝑦𝑛=2 )𝑦𝑛=1 = “ a b c ”

𝑦𝑛=2 =“ d e ”

ISR

𝑥𝑛=2 =𝑥𝑛=1 =ITTS

ITTS

𝑦𝑛=1 = “ a b c ”

𝑦𝑛=2 = “ d e ”

Full text = a b c d e f(𝑌)Copyright © 2021 NAIST Japan, All rights reserved

ISR

13

I.

Introduction

II. Incremental Machine
Speech Chain

III. Experiments

III. Experiments
IV. Conclusion

Copyright © 2021 NAIST Japan, All rights reserved

14Experiments

DatasetWall Street Journal CSR Corpus [Paul and Baker, 1992]• Language : English

Training sets:ooo

SI-84
SI-200SI-284: 16 hours of speech, 83 speakers: 66 hours of speech, 200 speakers: si84 + si200

Dev. set: dev93

Eval. set

: eval92• Character-level• Speech features: 80-dims log Mel spectrogram (window: 50 msec, shift: 12.5 msec)

Copyright © 2021 NAIST Japan, All rights reserved

15Experiments

Model Configuration* Same architecture for standard (non-incremental) and incremental models

ASR
y1

Transcription

y2

y3

TTS

yT

Tacotron 2 [Wang et al., 2017] structure with speaker embedding [Tjandra et al., 2018]

Decoder
CBHG

LSTM

Speech features

CharEmb.

x1,…, x4x5,…, x8xT-3, …, xT

Decodery0y1y2Linear Proj.

yT-1

2 LSTM

Attention
Encoder
Hierarchicalsub-sampling
8 featureframes into 1
encoderstate

Mel-to-linear

BiLSTM

x0BiLSTM

Input/step• ISR: 0.84 sec

BiLSTM• Std. ASR : full-utterance(avg. 7.88 sec)

FNNSpeech features

Speaker Emb.

2 Pre-Netx1 x2 x3 x4 x5 x6 x7 x8 xS-1 xS

x1,…, x4

Attention

Encoder
BiLSTM

Input/step• ITTS : max. 30 chars

3 Conv• Std. TTS : full-sentence(avg. 103 chars)

Char Emb.
Transcription

Copyright © 2021 NAIST Japan, All rights reserved

xT-7,…, xT-4y0

y1 y2

yT-1

16

Experiments

Learning Approach
Exploration on 2 learning approaches:

𝑥𝑛 =A) Semi-supervised incremental machine speech chain
1) ISR/ITTS independent training : supervised
2) ISR/ITTS joint training: unsupervised (unlabeled data)B) Supervised incremental machine speech chain
1) ISR/ITTS independent training : supervised
2) ISR/ITTS joint training : supervised (labeled data)

(B)

(A)

ITTS

ITTS

𝑦𝑛 = “ a b c ”“abc”

Unsupervised
GreedyISRSupervised
Teacherforcing

ISR
Correcttext

𝑥𝑛 =Unrolled process examples in joint training(ITTS-to-ISR follows similar mechanism)
Copyright © 2021 NAIST Japan, All rights reserved

17Result
ASR (CER%) and TTS (log Mel-spectrogram L2 loss) performances
TTS (L2-norm)2

ASR (CER%)(delay: 7.88 sec)

Incremental(delay: 0.84 sec)

Standard(delay: 103 chars)

Incremental(delay: 30 chars)

nat-spsyn-spnat-spsyn-spnat-txt

rec-txt

nat-txt

rec-txt

Indep-trn SI-84

17.33

27.03

17.81

44.54

0.99

1.02

1.04

3.62

Indep-trn SI-284

7.16

9.60

7.97

19.99

0.75

0.77

0.84

1.31

Data

Standard

Independent Training• Baseline
ISR and ITTS indep-trn SI-84• Topline
Standard systems indep-trn SI-284• Proposed
Incremental machine speech chain• Input type:nat-sp

ISR

Machine Speech Chain
rec-txt

Indep-trn (SI-84) +chain-trn-greedy (SI-200)

11.21

Indep-trn (SI-84) +chain-trn-teachforce(SI-200)

7.27

•

11.52

14.23

32.43

0.80

0.82

0.86

1.35

“”
ITTS

6.30

9.43

12.78

0.77

0.80

Incremental machine speech chaino Improved ISR and ITTS
o Shorter delay with a close performance to the standard system

Copyright © 2021 NAIST Japan, All rights reserved

0.79

1.26
nat-txt

ITTS

“”

ISR

“”

syn-sp18

I.

Introduction

II. Incremental Machine
Speech Chain

IV. Conclusion

III. Experiments
IV. Conclusion

Copyright © 2021 NAIST Japan, All rights reserved

19Conclusion
Incremental machine speech chain
Short-term feedback loop for ISR/ITTS development by mimicking human speech chaino Reduced the delay with a close performance to the basic frameworko Improve ISR and ITTS (natural/synthetic input)

Copyright © 2021 NAIST Japan, All rights reserved20

Thank you

Copyright © 2021 NAIST Japan, All rights reserved