IPSJ SIG Technical Report

Improving Intelligibility of Synthesized Speech in NoisyCondition with Dynamically Adaptive Machine SpeechChain
Sashi Novitasari1,2,a)

Sakriani Sakti1,2,3,b)

Satoshi Nakamura1,2,c)

Abstract: This paper focuses on the machine speech chain mechanism for improving the intelligibility of synthesized
speech in noisy conditions. Our proposed TTS synthesizes a speech by adapting to the situation. It will speak loudlywith high intelligibility in a noisy condition by processing auditory feedback that consists of speech-to-signal ratio
and ASR loss as a speech intelligibility measure. Our experiments show that auditory feedback improves the TTS in anoisy environment than the standard TTS.
Keywords: text-to-speech, machine speech chain inference, Lombard effect, dynamic adaptation

1.

Recent end-to-end text-to-speech synthesis (TTS) systems successfully produced natural-sounding speech given only the texttranscription [1], [2], [3], [4]. However, the standard systems suffer from speech intelligibility degradation in noisy places because
they only learn how to speak and do not learn how to listen andunderstand the situation.
In noisy places, humans speak louder to enhance their speech
audibility, a phenomenon known as the Lombard effect [5]. Lombard effect not only includes the change in the speech intensity but
also the pitch and speed [6]. Human speech adaptation, as in theLombard effect, is enabled by the speech chain mechanism [7], aclosed-loop auditory feedback mechanism from the mouth to theear. This feedback connection enables speakers to monitor theirspeech and improve the speech when necessary.
Inspired by the human speech chain mechanism, a machine
speech chain [8], [9], in Fig. 1(a), was proposed as a semisupervised learning method for ASR (ear) and TTS (mouth)by connecting both systems with a closed-feedback loop. This
framework improved ASR and TTS in training using unpaired
speech-text data. In inference, unfortunately, the feedback connection is removed, so the systems here are still unable to adapt
to their environment.
In this work, we propose a machine speech chain mechanism
for TTS inference in noisy places. Our TTS (Fig. 1(b)) synthesizes speech with a Lombard effect dynamically to improve123a)b)c)

TTS

Introduction

Nara Institute of Science and Technology, Ikoma-shi, 630–0192 JapanRIKEN, Center for Advanced Intel- ligence Project AIP, Ikoma-shi,630–0192 Japan
Japan Advanced Institute of Science and Technology, Nomi-shi, 923–1292 Japansashi.novitasari.si3@iss.naist.jp
ssakti@is.naist.jp
s-nakamura@is.naist.jp© 1992 Information Processing Society of Japan

TTS

text
Noise

text

text
LossASR+
SNR

text

Noisy speech

SNR-Emb

ASR(a)Fig. 1

text

ASR

(b)

(a) Previous machine speech chain for semi-supervised training; (b)proposed machine speech chain for training and dynamically adaptive inference.

the speech intelligibility given the auditory feedback in utterance
level. Here the auditory feedback consists of the speech-to-noise
ratio (SNR) as the speech and noise intensity measurement andthe ASR loss as the speech intelligibility measurement in noise.

2.

Proposed TTS in Speech Chain Framework

The proposed TTS, shown in Fig. 2(a), is a multi-speaker
Transformer TTS [2], [3] extended with auditory feedback components (ASR-loss embedding and SNR embedding) and a variance adaptor. Given character sequence x = [x1 , x2 , ..., xS ]with length S , TTS generates the corresponding speech Melspectrogram y = [y1 , y2 , ..., yT ] with a length T and the prosody
based on the auditory feedback in SNR (ZS NR ) and ASR loss
(ZAS R ) embedding. In noisy situations, the proposed TTS performs a dynamic adaptation with a feedback loop in several iterations until the ASR loss converges.
We construct three TTS systems with different feedback configurations. All systems are trained using normal speech andLombard speech in various noise conditions.
2.1 TTS with SNR feedback
TTS generates speech based on text input and SNR feedback
embedding. In noisy conditions, TTS re-synthesizes speech toachieve a higher SNR (≥ 20 dB) by integrating the SNR embedding into the encoder output and the decoder input. We imple-1

IPSJ SIG Technical Report

Encoder output
Stop token

Mel Linear

Variance
AdaptorNx

Stop Linear

SNR Label

Add & Norm

Softmax

LinearFNN

Intensity Predictor

LinearLinearAdd & Norm

Multi-head
AttentionFNNEncoderResBlock

Add & NormNx

Add & Norm

Add & Norm

Multi-head
Attention

MaskedMulti-head
Attention

Decoder

ASR Loss
Emb.

ASR++
TTS speech

DecoderPre-netPositional
Encoding++Noise

Noisyspeech"spare"Encoder input

Duration Predictor

Noisy
speech"spare"SNR
Emb.

EncoderPre-netPitch Predictor

Nx

Conv + ReLU

TTS input text

"space"Speaker
Emb.
Positional
Encoding

ResBlock

Nx

Conv + ReLU

+Feedback input

TTS speech

Noise

Decoder input(a) Proposed TTS

Fig. 2

Table 1(b) ASR-loss Embedding

Speech intelligibility measure (CER %) at different SNR levels using clean- and multi-condition training ASR.
SystemMulti-condition training ASRClean
SNR 0
SNR -10Baseline TTS
Standard TTS
18.32+ Rule-based modification into Lombard speech
18.32+ Fine tuning with Lombard speech (SNR 0)
13.19+ Fine tuning with Lombard speech (SNR -10)
14.26
+ Fine tuning with Lombard speech (SNR 0 + SNR -10)
13.40
Proposed TTS
TTS in speech chain framework
18.32+ SNR feedback
11.58+ SNR-ASR feedback
12.55+ SNR-ASR feedback + variance adaptor
11.99
Topline (human natural speech)
Natural speech
7.43+ Rule-based modification into Lombard speech
7.43
Natural Lombard speech
7.43

70.54
44.68
32.71
24.47
28.12

77.07
57.86
53.35
40.62
46.13

70.54
22.82
16.11
14.70

77.07
42.00
25.61
24.96

22.17
13.24
11.46

58.81
15.15
20.46

ment the SNR embedding module using convolution network layers (Fig. 2(c)), which generate the embedding ZS NR from noisy
speech features ynoisy .
2.2 TTS with SNR-ASR feedback
TTS synthesizes speech based on text input and feedback fromthe SNR and ASR-loss embedding. The ASR-loss embedding(Fig. 2(b)) represents the speech intelligibility measurement in
the presence of noise. ASR-loss embedding vector ZAS R is generated by transcribing the noisy TTS speech using ASR and feeding
the ASR recognition loss to the embedding module.
2.3 TTS with SNR-ASR feedback and variance adaptor
In addition to the SNR and ASR-loss embedding feedback,the proposed TTS applies a variance adaptor [10] that guides theprosody adaptation by predicting the speech intensity, duration,and pitch (Fig. 2(d)). The prosody attributes are predicted fromthe encoder Transformer output fused with feedback embeddings.
The module output is utilized to produce the encoder final output.

3.

(c) SNR Embedding(d) Variance Adaptor

Architecture: (a) proposed TTS with a Transformer-based encoder-decoder structure, extended
with (b) ASR-loss embedding, (c) SNR embedding, and (d) variance adaptor [10] modules.

Experiment

We experimented on using Wall Street Journal (WSJ) corpus[11]. Here we also recorded natural Lombard speech in noisy© 1992 Information Processing Society of Japan

conditions with a single male speaker. The noises in the recording were simulated by generating noises of SNR 0 dB and SNR-10 dB based on WSJ clean speech data first. From the prosody
attributes changes observed in the recorded Lombard speech, weconstructed synthetic Lombard WSJ speech by modifying theoriginal WSJ speech pitch, duration, and intensity into a target
SNR 20 dB. The original WSJ speech and the synthetic Lombard
WSJ speech were used for TTS training and testing. Here ourbaselines are (1) the standard TTS, (2) the standard TTS with therule-based speech modification into the Lombard speech, and (3)the standard TTS fine-tuned to Lombard speech [12]. The rulebased speech modification into Lombard speech was the same
method as the synthetic Lombard WSJ construction method. The
topline is the natural human speech.
The proposed SNR feedback and ASR feedback mechanism
significantly improved the TTS speech intelligibility in noisy conditions, shown in Table 1. SNR feedback made the TTS awareof the environmental noise and speak with the Lombard effect,and ASR feedback helped the TTS improve the speech intelligibility further. Here the variance adaptor guided the speechprosody adaptation and resulted in a better intelligibility enhancement. The results here reveal that the machine can also dynamically adapt in several loops; listen to its voice in a noisy
environment and then speak louder to improve it. For further information on speech samples, see the following reference:https://sites.google.com/view/lombard-dynamic-tts/home.

4.

Conclusions

We constructed a dynamically adaptive machine speech chain
inference framework to improve TTS intelligibility in noisy conditions. Our results reveal that dynamic adaptation with auditory
feedback is important not only for humans but also for machines
to generate a highly audible speech in various conditions.
Acknowledgments Part of this work is supported by JSPS
KAKENHI Grant Numbers JP17H06101 and JP21H03467.

IPSJ SIG Technical Report

References
[1]

[2]
[3]
[4]
[5]
[6]
[7]

[8]
[9]
[10]
[11]

[12]

Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly,
Z. Yang, Y. Xiao, Z. Chen, S. Bengio et al., “Tacotron: Towards endto-end speech synthesis,” in Proc. INTERSPEECH, 2017, pp. 4006–4010.
N. Li, S. Liu, Y. Liu, S. Zhao, and M. Liu, “Neural speech synthesis
with Transformer network,” in Proc. AAAI Conference on Artificial
Intelligence, vol. 33, no. 01, 2019, pp. 6706–6713.
M. Chen, X. Tan, Y. Ren, J. Xu, H. Sun, S. Zhao, and T. Qin, “MultiSpeech: Multi-speaker text to speech with Transformer,” in Proc.
INTERSPEECH, 2020, pp. 4024–4028.
J. Donahue, S. Dieleman, M. Bińkowski, E. Elsen, and K. Simonyan,“End-to-end adversarial text-to-speech,” in Proc. ICLR, 2021.
H. Lane and B. Tranel, “The Lombard sign and the role of hearing inspeech,” Journal of Speech and Hearing Research, vol. 14, no. 4, pp.
677–709, 1971.
T. Letowski, T. Frank, and J. Caravella, “Acoustical properties ofspeech produced in noise presented through supra-aural earphones,”
Ear and hearing, vol. 14, pp. 332–338, 1993.
P. B. Denes and E. N. Pinson, The Speech Chain: The Physics
and Biology of Spoken Language, ser. Science/communication.
New York, N.Y: W.H. Freeman, 1993. [Online]. Available:
https://books.google.co.jp/books?id=UBtiAAAAMAAJ
A. Tjandra, S. Sakti, and S. Nakamura, “Listening while speaking:Speech chain by deep learning,” in Proc. IEEE ASRU, 2017, pp. 301–308.
——, “Machine speech chain,” IEEE/ACM Transactions on Audio,Speech, and Language Processing, vol. 28, pp. 976–989, 2020.
Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Liu, “FastSpeech 2: Fast and high-quality end-to-end text to speech,” ArXiv, vol.
abs/2006.04558, 2020.
D. B. Paul and J. M. Baker, “The design for the Wall Street Journalbased CSR corpus,” in Proceedings of the workshop on Speech andNatural Language. Association for Computational Linguistics, 1992,pp. 357–362.
D. Paul, M. P. Shifas, Y. Pantazis, and Y. Stylianou, “Enhancingspeech intelligibility in text-to-speech synthesis using speaking styleconversion,” in Proc. INTERSPEECH, 2020, pp. 1361–1365.

© 1992 Information Processing Society of Japan3