Improving Intelligibility of Synthesized Speechin Noisy Condition with
Dynamically Adaptive Machine Speech ChainzSashi Novitasari1,2, Sakriani Sakti 1,3,2, and Satoshi Nakamura 1,21Nara Institute of Science and Technology, Japan

2RIKEN, Center for Advanced Intelligence Project (AIP), Japan
3Japan Advanced Institute of Science and Technology, Nomi-shi, 923– 1292 Japan

E-mail: {sashi.novitasari.si3, ssakti, s-nakamura}@is.naist.jp

Copyright © 2021 NAIST Japan, All rights reserved

SPEECH PRODUCTION

How about humans?

State-of-the-art: End-to-end neural TTS• Synthesizes a human-like speech in

In noisy situation, we tend to speak louder(Lombard effect)
noiseclean condition“Hello”

HowTTS• Noisy condition?
“Hello”

are you?

noisenoiseTTS

Cannot perform well!
Copyright © 2021 NAIST Japan, All rights reserved

noisenoise● Existing work with neural TTS:
Fine-tuning to certain noise [Paul et al., 2020]● Human:No fine-tuning before speaking in noisy placeà How?
2

SPEECH PRODUCTION
HumanTTS● Humans speak while listen to their own speech
Speech chain[Denes, 1993]● Computers only learn how to speak● Cannot hear their own voicenoise

How are you?

How are you?

Auditory feedback

Speakingnoise

Copyright © 2021 NAIST Japan, All rights reserved

Adapt to the situation

Auditory feedback

Speaking
Only speaking
3

MACHINE SPEECH CHAIN●●

Introduced in 2017 [Tjandra et al., 2017]
ASR and TTS are connected via closed feedback
loop during trainingà Support each other and improve together

Limitation: Only for training mechanism

Machine speech chain

unroll

ASR→ TTS (speech only)● In inference, ASR and TTS perform separately asin the standard manner● Unable to dynamically adapt based on various
conditions (unlike humans)

𝐿𝑜𝑠𝑠𝑇𝑇𝑆

TTS → ASR (text only)

text

𝐿𝑜𝑠𝑠𝐴𝑆𝑅

TTS

ASR

text
TTS

updateASR

updatetext
Copyright © 2021 NAIST Japan, All rights reserved

4

PROPOSED METHODNew Generation of Machine Speech Chain
Dynamically Adaptive Machine Speech Chain Inference for TTS
TTS speaks louder in noisy environment by taking auditory feedback

TTS

text

Variance
adaptor

LossASRtext

noise+noisy speech

SNR

Auditory feedback

SNR-Emb

ASR

Copyright © 2021 NAIST Japan, All rights reserved

5RELATED WORKSTTS IN NOISY CONDITION
Parametric TTS in noise• HMM TTS speech modification to increase speech intelligibility in noise
while keeping the speech energy fixed [Valentini-Botinhao et al., 2014;
Schepker et al., 2015]• HMM TTS adapted to Lombard speech data [Raitio et al., 2014]
Neural network-based TTS in noise• Transfer learning from a standard end-to-end TTS (clean) to an end-toend Lombard TTS [Paul et al., 2020]o Lombard TTS is trained on a small Lombard dataset• End-to-end multi-style TTS [Hu et al., 2021]
o Synthesizable speech styles: Normal speech, whispered speech,
Lombard speech

Offline fine-tuning

Our focus
End-to-end Lombard TTS withdynamic adaptation using auditory
feedback, similar to human
6

PROPOSED METHOD

Copyright © 2021 NAIST Japan, All rights reserved

7

PROPOSED TTS

TRANSFORMER TTS WITH AUDITORYFEEDBACK●Basic TTS structure: Transformer TTS [Li et el., 2018]○ Input : Characters○ Output : Speech features (80 dims. Mel-spectrogram)○○●Multi-speaker experiment: Multi-speaker TTS
Transformer [Chen et al., 2020]
Speaker embedding: Deep Speaker [Li et al., 2017](similar to TTS in the basic machine speech chain)

Proposed TTS structure:a) TTS + SNR embeddingb) TTS + ASR-SNR embeddingc) TTS + ASR-SNR embedding + Variance adaptor

Copyright © 2021 NAIST Japan, All rights reserved

Multi-speaker Transformer TTS with Deep
Speaker speaker embedding

8

A. TTS with SNR embedding
Auditory feedback• SNR embedding(𝑍!"# ): SNR of noisy speech (𝑦 $%&'()

𝑍!"# = 𝑆𝑁𝑅 𝐸𝑚𝑏 (𝑦 $%&'( )o Trained as SNR recognition model firsto Utilized in:§ Encoder output (ℎ) ))ℎ) = ℎ*+,+ 𝑍!-. + 𝑍!"#&§ Decoder first layer Input (𝑦*+,)&𝑦*/0= 𝑝𝑟𝑒𝑛𝑒𝑡 𝑦*/0 + 𝑍!-. + 𝑍!"# + 𝑃𝐸𝑍!"# : speaker embedding
𝑃𝐸 : positional encoding
SNR emb. module
Copyright © 2021 NAIST Japan, All rights reserved

Transformer TTS with SNR emb.

9

B. TTS with SNR and ASR-loss embedding
Auditory feedback:• SNR embedding• ASR-loss embedding (𝑍-!# ): Maps the ASR MSE
loss into embedding space

𝑍-!# = 𝐴𝑆𝑅 𝐿𝑜𝑠𝑠 𝐸𝑚𝑏 𝐿𝑜𝑠𝑠-!# 𝑥, 𝑝.
𝑝. = 𝑝(𝑥|𝑦 $%&'( )𝑥 : TTS input text (correct text)
𝑝$ : ASR hypothesis•

Utilized in encoder output and decoder input:)ℎ) = ℎ*/0+ 𝑍!12 + 𝑍!"# + 𝑍-!#&𝑦*+,= 𝑝𝑟𝑒𝑛𝑒𝑡 𝑦*+, + 𝑍!12 + 𝑍!"# + 𝑍-!# + 𝑃𝐸

ASR-loss emb. module
Copyright © 2021 NAIST Japan, All rights reserved

Transformer TTS with SNR andASR-loss emb.

10

C. TTS with SNR, ASR-loss embedding, and variance adaptor
Auditory feedback:• SNR embedding• ASR-loss embedding
Prosody guide: Variance adaptor• Based on variance adaptor in Fast Speech [Ren et al., 2020],
modified for autoregressive Transformer decoder• 3 components à predict character-level speech prosody:)𝑣 3 = 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑜𝑟 3 (ℎ*/0+ 𝑍!12 + 𝑍!"# + 𝑍-!# )o Intensity predictor (X = G)o Pitch predictor (X = P)o Duration predictor (X = D)

• Add the speech prosodies information to encoder output :)ℎ) = 𝑣 4 + 𝑣 1 + 𝑣 5 + (ℎ*/0+ 𝑍!12 + 𝑍!"# + 𝑍-!# )Copyright © 2021 NAIST Japan, All rights reserved

Variance adaptor

Transformer TTS with SNR, ASR-loss
embedding, and variance adaptor

11Experiments

Copyright © 2021 NAIST Japan, All rights reserved

12

EXPERIMENT SETTINGDATA
A. Clean Wall Street Journal (WSJ) speech [Paul et al., 1992]• Multi-speaker English speech, 81 hours of speech• Training: SI-284 set, dev: dev92 set, test: eval93 set

clean speech (A)+B. WSJ speech with additive noise• Clean WSJ speech combined with noisy soundo Noise type : white noise and babble noiseo SNR: SNR 0 and SNR -10
C. Natural Lombard speech• Clean and noisy speech recorded from single male speaker• Text: WSJ speech transcription (dev92 + eval93)
D. Synthetic Lombard WSJ speech• Clean WSJ speech with the intensity, pitch, and duration
modified into Lombard speech
Copyright © 2021 NAIST Japan, All rights reservednoise

noisy speech (B)Lombard speech(noise))ise(no)(noise13

EXPERIMENT SETTINGSYSTEM CONFIGURATION
Topline: Natural Lombard speech
Models structure and training data configuration
SystemStructure

Training Data

TTS
Baseline standard TTS
Baseline standard TTS+ Fine-tuning [Paul et al., 2020]

Clean WSJ
Transformer- 6 Enc, 6 Dec

Proposed TTS

Clean WSJ + Synthetic Lombard WSJ
Clean WSJ + Synthetic Lombard WSJFeedback component
ASR
SNR recognition
Copyright © 2021 NAIST Japan, All rights reservedTransformer- 12 Enc, 6 Dec(Speech-transformer [Dong et al., 2018])

4 convolutional + residual layers

Clean WSJ + Noisy WSJ
Clean WSJ + Noisy WSJ(class: clean, SNR 0, SNR -10)

14

RESULT● Evaluation à Speech intelligibility
metric:o ASR Character error rate (CER)o ASR recognize noisy TTS speech● Proposed TTS max. feedback loop: 4● Best performance by TTS + SNR-ASR
loss emb. + variance adaptor
o SNR and ASR feedback improved
the speech intelligibilityo Variance adaptor guided theprosody change well by providing
the target prosody information

How the auditory feedback
affected the TTS performance?

Speech intelligibility measure (CER %) at different SNR levels
using ASR trained on clean and noisy conditions.
SystemClean
SNR 0
SNR -10Baseline TTS
Standard TTS

70.54

77.07

+

modification into Lombard speech

18.32

44.68

57.86

+

Fine-tuning with Lombard speech

13.40

28.12

46.13

Proposed TTS
TTS + SNR emb.

11.58

22.82

42.00

TTS + SNR-ASR loss emb.

12.55

16.11

25.61

TTS + SNR-ASR loss emb. + var. adaptor

11.99

14.70

24.96

7.43

22.17

58.81

7.43

13.24

15.15

7.43

11.46

20.56

Topline (human natural speech)
Natural speech+modification into Lombard speech

Natural Lombard speech
Copyright © 2021 NAIST Japan, All rights reserved

18.32

15

Result

How the auditory feedback affects TTS speech?
● Experiments by applying a coefficient to SNR embedding and ASR-loss embedding in encoder

output and decoder input (default coefficient: 1)

The effect of auditory feedback on speech intelligibility

best!

● Clean condition: best performance with ASR feedback only (ASR coeff 1, SNR coeff 0)● Noisy condition: best performance by equal amount of ASR + SNR feedback (coeff 1)

Both SNR and ASR-loss information are important to synthesize Lombard speech
Copyright © 2021 NAIST Japan, All rights reserved16

Result

How the feedback loop affects TTS speech?
The effect of feedback loopon speech intelligibility● Loop 1 : No feedback utilization● Improvement significantly occursafter the 2nd loop

TTS performed dynamic adapt in several loops;
listen to its voice in a noisy environment andthen speak louder (similar to humans)
Copyright © 2021 NAIST Japan, All rights reserved

17

CONCLUSION● Dynamically adaptive machine speech chain inference framework to support TTS in noisy
conditions.
● The proposed systems with auditory feedback and a variance adaptor produced a highly
intelligible speech that surpassed a standard TTS with a fine-tuning method and achieved
closer to the human performances.
● Dynamic adaptation with auditory feedback is critical not only for human but also in speech
generation by machines

Copyright © 2021 NAIST Japan, All rights reserved

18

THANK YOU

Copyright © 2021 NAIST Japan, All rights reserved

19Appendix

20

RESULT● Evaluation à Speech intelligibility
metric:o ASR Character error rate (CER)o ASR recognize noisy TTS speech● Proposed TTS max. feedback loop: 4● Best performance by TTS + SNR-ASR
loss emb. + variance adaptor
o SNR and ASR feedback improved
the speech intelligibilityo Variance adaptor guided theprosody change well by providing
the target prosody information

How the auditory feedback
affected the TTS performance?

Speech intelligibility measure (CER %) at different SNR levels
using ASR trained on clean and noisy conditions.
SystemClean
SNR 0
SNR -10Baseline TTS
Standard TTS

70.54

77.07

+

modification into Lombard speech

18.32

44.68

57.86

+Fine-tuning with Lombard speech

13.40

28.12

46.13

Proposed TTS
TTS + SNR emb.

11.58

22.82

42.00

TTS + SNR-ASR loss emb.

12.55

16.11

25.61

TTS + SNR-ASR loss emb. + var. adaptor

11.99

14.70

24.96

7.43

22.1758.81

7.43

13.24

15.15

7.43

11.46

20.56

Topline (human natural speech)
Natural speech+

modification into Lombard speech

Natural Lombard speech
Copyright © 2021 NAIST Japan, All rights reserved

18.32

21Speech intelligibility measure (CER %) at different SNR levels using
clean- and multi-condition training ASR
SystemStandard TTS

Clean Condition Training ASR

Multi-condition Training ASR

Clean
SNR 0
Baseline TTS

SNR -10Clean

SNR 0

SNR -1018.92

118.72106.25

18.32

70.54

77.07+

modification into Lombard speech (rule)

18.92

102.96104.6918.32

44.6857.86+

Fine-tuning with Lombard speech (SNR0)

10.76

93.19

105.01

13.19

32.71

53.35

+

Fine-tuning with Lombard speech (SNR-10)

11.73

71.88

99.36

14.26

24.47

40.62

+

Fine-tuning with Lombard speech (SNR0 +SNR-10)

11.25

79.94

100.44

13.40

28.12

46.13

Proposed TTS
TTS + SNR emb

10.21

83.15

101.41

11.58

22.82

42.00

TTS + SNR-ASR loss emb.

10.76

52.51

87.72

12.55

16.11

25.61

TTS + SNR-ASR loss emb. + variance adaptor

10.47

55.70

92.75

11.99

14.70

24.96

Topline (human natural speech)
Normal speech+

modification into Lombard speech (rule)

Lombard speech

5.77

92.56

98.98

7.43

22.17

58.81

5.77

58.40

67.78

7.43

13.24

15.15

5.77

25.38

59.25

7.43

11.46

20.56

22

TTS with SNR, ASR-loss embedding, and variance adaptor
Variance adaptor• Predictor training loss
𝑣1 = predicted prosody
𝑣 = prosody label
𝑆 = character seq. length• Label: character-level prosodyooChar-speech alignment: Force-alignment
Prosody label: extracted using FastSpeech open-source code

• TTS training loss

𝑇 = speech length𝑦=
1 pred. speech𝑦= ref. speech
6 pred. stop token
𝑏=𝑏 = stop token label

𝑣1 ! = pred. pitch𝑣 " = ref. pitch
𝑣1 # = pred. intensity
𝑣 # = ref. intensity

𝑣1 $ = pred. duration𝑣 $ = ref. duration

Variance adaptor

Transformer TTS with SNR, ASR-loss
embedding, and variance adaptor

23

DATA PREPARATION (3)D. Synthetic Lombard WSJ speech• Clean WSJ speech with the modified prosody
o Intensity increased to reach SNR 20
o Pitch/duration were increased using a coefficient based on speech phoneme-level
pitch/duration changes in natural Lombard speech (dev92) to keep speaker characteristic

Speech examples (noise: from SNR -10)
A. Clean WSJ

B. Clean WSJ +noise

C. NaturalLombard speech

D. Synthetic Lombard WSJ
clean
noisy

24