Self-adaptive Incremental Machine Speech Chain for Lombard TTS
with High-granularity ASR Feedback in Dynamic Noise Condition
Sashi Novitasari1, Sakriani Sakti2,1, and Satoshi Nakamura1
1Nara Institute of Science and Technology, Japan; 2Japan Advanced Institute of Science and Technology, Japan

I. BACKGROUND
A. TTS in noisy place
HumanStandard TTS
How are you?

How are you?

Intelligibility
degrade!

TTS
noise

Auditory feedback

Listeningnoise

Speaking

Speaking

Only speaking

III. EXPERIMENT

Adapt-ITTS withhigh-granularity ASR feedback

A. Setting

Adapt to the situation

Louder, slower, higher pitch, etc.

● Noise degrades the TTS speech intelligibility ● Speech chain. Speaking while listening● TTS is commonly trained using speech data
[Denes and Pinson, 1993]
from the clean environment● Lombard effect. Speaking effort● Common solution: Fine-tune the TTS using
adjustment in noisy place to improve
Lombard speech data (static noise)
the speech intelligibility[Raitio et al., 2011; Paul et al., 2020]● For each incremental step, use the charactervocabulary level ASR feedback based on thelosses of the positive and negative classes● ASR feedback is an ASR loss embedding (zASR)● Character-level ASR loss
● Generated by transcribing noisy TTS speechusing an ASR● POW, synth. speech power

ASR

Architecture
Encoder output

Incremental step 1
Stop token

HOW

ARE

LinearTTS

Synthesized speech+ noiseTransformer

Incremental step 2

ASR
SNR

Feedback
modulePOW

Feedback

TTS

ASR
SNR

Feedback
modulePOW

Feedback

Speaker
Emb.

EncoderPre-netAdapt-ITTS

SNR
Emb.

FNN

Standard or
non-adaptive TTSCASR Loss
Emb.

DecoderPre-net"how"speech+noise

Positional
Encoding

Encoder input
Feedback input

Decoder input

LossASR =loss(x𝑙+,+(𝑥+ =1=A)“𝐴”)𝑙loss(x“𝐴”).,+ (𝑥. =2=A)loss(x“𝐻”)
𝑙+,(𝑥+ =1=H)“𝐻”) 𝑙loss(x.,- (𝑥. =2=H)...

loss(x=O)
𝑙+,+/(𝑥+ 1==O)“𝑂”) 𝑙loss(x“𝑂”).,+/ (𝑥. 2=...

loss(x=U)
𝑙+,.+(𝑥+ 1==U)“𝑈”) 𝑙loss(x“𝑈”).,.+ (𝑥. 2=

Decoder

Positional
Encoding

23𝒚()*!+
Synthesized speech+ noise92

loss(x2=B)

loss(x1=H)88 loss(x2=H)

...

21

proposed84

loss(x1=O) loss(x2=O)

...

19(a)
Hurricane80(a)Limitation:● Low-granularity ASR loss feedback only basedon positive character class (limited information)● Training based on synthetic Lombard data...

From“A” to “Z”

From“A” to “Z”

(b)

...

TTS speech ASR CER ↓Low-granularity ASR feedback[Previous work]
x1

proposed... loss(x =A)
loss(x =B)

ASR𝑙loss(x“𝐻”)
0,- (𝑥0 =S=H)

Noise:
loss(xS=H)● Babble noise and white noise● SNR: -10 dB, 0 dB, clean (no noise)loss(xS=O)
Model:
loss(xS=U)
Baseline
Finetune-TTS

TTS speech STOI↑83proposed

Adapt-ITTS
With ASR feedback: low granularity
With ASR feedback: high granularity(proposed)

79

zASR

75

The proposed high-granularity ASR feedback improved theASR Loss Emb.
incremental TTS speech intelligibility

LossASR =LossASR =“𝑈”)𝑙!loss(x(𝑥! =1=H)“𝐻”) 𝑙loss(x“𝑂”) ... 𝑙loss(x# (𝑥# =S=U)" (𝑥" =2=O)

loss(x1=A)

Only “O”

S

intelligibility measure.

81

24

ASR Loss Emb.

Only “H”

proposedloss(x =B)

77

xSCorrectHOW
ARE YOU𝑙(𝑥=“𝐴”)
0,+0
Stext
ASRHOU ARE YOO𝑙(𝑥output
0,. 0 =S “𝐵”)(b)
85

26zASR...

(incremental)

Decoder Mel-spectrogram output

Power ContextEmb.

...

30

ASR Loss Emb.

loss(x“𝐵”)𝑙+,.
(𝑥+ =1=B)“𝐵”) 𝑙loss(x.,. (𝑥. =2=B)

Mechanism

Linearloss(x1=B)

28

ASR
Loss
= and feedback modules:
Autoregressive
Transformer-based
TTS withvariance
HOU
ARE YOO
ASRadaptor
output● ASR loss, based on the noisy synth. speechloss(x1=H) loss(x2=O) ... loss(xS=U)● SNR, speech-to-noise ratio

Transformer
EncoderASRxSVariance
Adaptor9625zASR

Correct
End-to-end
incremental
TTS (ITTS) that adapts the speaking style using theHOWARE YOU
ASR
LossEmb.
text
auditory feedback based on the prev. synthesized speech + noise+ASR Loss Emb.
ASR Loss Emb.
WSJ
Positive
TTS speech intelligibility measure:
Loss=
LossASR =charASR
seq.
TTS speech ASR CER ↓TTS speech STOI↑● ASR CER (%) : Character error rate.
loss(x1=H) loss(x2=O) ... loss(xS=U)
loss(x1=A) loss(x2=A) ... loss(xS=A) ● STOI (%): Short-term objective

Proposed ASR feedback generation methodzC

B. Result: TTS speech intelligibility in noisy situation

xSCorrect
HOW ARE YOUtextASR
HOU ARE YOO
output𝑧#$% = 𝐴𝑆𝑅 𝐿𝑜𝑠𝑠 𝐸𝑚𝑏𝑒𝑑𝑑𝑖𝑛𝑔 (𝐿𝑜𝑠𝑠#$% (𝒙, 𝒑𝒙 ))𝒑' = 𝑝#$% (𝒙|𝒚()*!+ )B. Adapt-ITTS: Self-adaptive incremental TTS withASR
machine speech chain mechanism [Novitasari et al., 2022]

FNN

x1zASRzASR

loss(x1=U) loss(x2=U)

TTS limitation:● No auditory feedback mechanism● Cannot self-adapt to noisy situation

x1● Training data:o WSJ [Paul & Baker, 1992]: Natural normal speech + synthetic Lombard speech (multispeaker)o Hurricane [Cooke et al., 2013]: Natural normal speech + natural Lombard speech(single speaker)● Architecture: Autoregressive transformer + variance adaptor + feedback modules

Improve the Adapt-TTS by enriching the ASR auditory feedback
information

noise

noise

II. PROPOSED METHOD

Only “U”

loss(x1=B)...

loss(x2=A)... loss(xS=A)

IV.
loss(xCONCLUSION=B)
loss(xS=B)
2loss(x2=H)
loss(x• MCE loss for each character timestep (s) loss(x1=H) Adapt-ITTS
with
the high-granulated ASR feedback for the selfS=H)...
• For positive character class only

adaptiveloss(x2=O) speechloss(xS=O)synthesis in noisy conditions
Synthesized𝑙loss(x==O)“𝑈”) speech 𝑙! 𝑥! , 𝑝" = − ∑&#$% 𝟙 𝑥! = 𝑐 ∗ log 𝑝" 𝑐 ...
0,+/ (𝑥0 S+ noise● Adapt-ITTS adapts the speech style based on noise conditions
loss(x1=U) loss(x2=U)
loss(xS=U)● Short-term feedback in an incremental mechanism...
𝑙loss(x==U)“𝑈”)(a)(b)0,.+ (𝑥0 S● The proposed ASR feedback improved Adapt-ITTS intelligibility
Positive character class (𝑥!" = 1)
innoisy
conditions
From"loss(x1=O)!

“A” to “Z”!

Negative character class (𝑥! = 0)

Scan for speech samples● BCE loss for each character (c) in dictionary for each character timestep(s)𝑙!,# 𝑥!# , 𝑝$! = −(𝑥!# ∗ log 𝑝$! 𝑐 + 1 − 𝑥!# ∗ log(1 − 𝑝$! 𝑐 ))or
https://sites.google.com/view/adapt-lombard-tts/home